- WLUG - wlug.mailman3.com

Re: [Wlug] large file performance on linux

by Klein, Richard

> -----Original Message----- > From: Jeff Moyer > > p.s. In your head, is Mb Megabit or Megabyte? I always thought that lower-case b meant bit and upper-case B meant byte, but the rest of the world doesn't seem to care what I think. -- Rich

May 16, 2007

Re: [Wlug] large file performance on linux

by brad noyes

On Wed, May 16, 2007 at 11:44:13AM -0400, Jamie Guinan wrote: > On Wed, 16 May 2007, brad noyes wrote: > Hi, > > Do you mean MB/s as in MegaBytes/second? Lower-case "b" implies > "bits" to me, but I suspect you meant bytes. :) > Yes, your assumption is correct. I never quite understood the capitalization conventions when it came to computers. > Anyway, if you know your total data set is going to exceed your system > memory, which will largely get used as page cache, you might as well > open the output with O_SYNC and write straight out. Then the big > buffer in the writer thread can go away, and you can just queue chunks > between the input and writer. > > A good experiment would be to run the writer thread with dummy input > data and see what kind of throughput you get. > > You could try ext2 (no journalling), or tweaking some of the > journalling options in ext3 (data=journal/ordered/writeback, see "man > mount"). > great idea. I forgot about tune2fs and mount options. thanks. > If you have enough CPU bandwidth, and your input stream has enough > redundancy, you could gzip it before writing, which might reduce your > output bandwidth requirements. > There isn't enough redundancy in the data. its essentially random data so it won't compress well at all. > Hope this helps, keep us posted. > Thanks for the input. -- brad

May 16, 2007

Re: [Wlug] large file performance on linux

by Jamie Guinan

On Wed, 16 May 2007, brad noyes wrote: > Hello All, > > I am seeing some really slow performance regarding large files on linux. I > write a lot of data points from a light sensor. The stream is about 53 Mb/s and > i need to keep this rate for 7 minutes, that's a total of about 22Gb. I > can sustain 53Mb/s pretty well until the file grows to over 1Gb or so, then > things hit the wall and the writes to the filesystem can't keep up. The writes > go from 20ms in duration to 500ms. I assume the filesystem/operating system > is caching writes. Do you have any suggestions on how to speed up performance > on these writes, filesystem options, kernel options, other strategies, etc? > > Things I have tried: > - I have tried this on a ext3 file system as well as an xfs filesystem > with the same result. > > - I have also tried spooling over several files (a la multiple volumes) > but i see no difference in performance. In fact, i think this actually > hinders performance a bit. > > - I keep my own giant memory buffer where all the data is stored and > then it is written to disk in a background thread. This helps, but > i run out of space in the buffer before i finish taking data. Hi, Do you mean MB/s as in MegaBytes/second? Lower-case "b" implies "bits" to me, but I suspect you meant bytes. :) Anyway, if you know your total data set is going to exceed your system memory, which will largely get used as page cache, you might as well open the output with O_SYNC and write straight out. Then the big buffer in the writer thread can go away, and you can just queue chunks between the input and writer. A good experiment would be to run the writer thread with dummy input data and see what kind of throughput you get. You could try ext2 (no journalling), or tweaking some of the journalling options in ext3 (data=journal/ordered/writeback, see "man mount"). If you have enough CPU bandwidth, and your input stream has enough redundancy, you could gzip it before writing, which might reduce your output bandwidth requirements. Hope this helps, keep us posted. -Jamie > > Thanks, > -- Brad > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug >

May 16, 2007

Re: [Wlug] large file performance on linux

by Jeff Moyer

==> On Wed, 16 May 2007 14:53:16 +0000, brad noyes <maitre(a)ccs.neu.edu> said: brad> Hello All, brad> I am seeing some really slow performance regarding large files on linux. I brad> write a lot of data points from a light sensor. The stream is about 53 Mb/s and brad> i need to keep this rate for 7 minutes, that's a total of about 22Gb. I brad> can sustain 53Mb/s pretty well until the file grows to over 1Gb or so, then brad> things hit the wall and the writes to the filesystem can't keep up. The writes brad> go from 20ms in duration to 500ms. I assume the filesystem/operating system brad> is caching writes. Do you have any suggestions on how to speed up performance brad> on these writes, filesystem options, kernel options, other strategies, etc? Of course. Your data set is larger than the page cache, so when you hit the low watermark, it starts write-back. You can deal with this a few different ways, and I'll throw out the easiest ways first: 1) Get more memory 2) Get a faster disk If those are not options, then you can tweak your application by using AIO and O_DIRECT. This will allow you to drive your disk queue depths a bit further and avoid the page cache. Check the man pages for io_setup, io_submit, and io_getevents to get started. brad> Things I have tried: brad> - I have tried this on a ext3 file system as well as an xfs filesystem brad> with the same result. You may not want to use a journalled file system. If you must, though, with ext3 you could try running with the data=writeback option. brad> - I have also tried spooling over several files (a la multiple volumes) brad> but i see no difference in performance. In fact, i think this actually brad> hinders performance a bit. I'm not sure I fully understand what you mean. Are you saying you write to separate physical volumes, and that you don't see any performance increase from doing so? brad> - I keep my own giant memory buffer where all the data is stored and brad> then it is written to disk in a background thread. This helps, but brad> i run out of space in the buffer before i finish taking data. Right, this is exactly what happens in the OS. ;) Speaking of which, you don't mention which kernel you are using. Could you please provide that information? There are a few vm tunables that you could try tweaking, but I really don't think they will help if your data set is larger than memory. We can explore that option, though, if you like. For now, my suggestion is to try using AIO with the open flag O_DIRECT. This will require you to align your data on 512 byte boundaries (and the size of the I/Os has to be a multiple of 512 as well). If you need any help converting your app, feel free to contact me off-list. -Jeff p.s. In your head, is Mb Megabit or Megabyte?

May 16, 2007

large file performance on linux

by brad noyes

Hello All, I am seeing some really slow performance regarding large files on linux. I write a lot of data points from a light sensor. The stream is about 53 Mb/s and i need to keep this rate for 7 minutes, that's a total of about 22Gb. I can sustain 53Mb/s pretty well until the file grows to over 1Gb or so, then things hit the wall and the writes to the filesystem can't keep up. The writes go from 20ms in duration to 500ms. I assume the filesystem/operating system is caching writes. Do you have any suggestions on how to speed up performance on these writes, filesystem options, kernel options, other strategies, etc? Things I have tried: - I have tried this on a ext3 file system as well as an xfs filesystem with the same result. - I have also tried spooling over several files (a la multiple volumes) but i see no difference in performance. In fact, i think this actually hinders performance a bit. - I keep my own giant memory buffer where all the data is stored and then it is written to disk in a background thread. This helps, but i run out of space in the buffer before i finish taking data. Thanks, -- Brad

May 16, 2007

Re: [Wlug] PGP Key Signing Party

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Eric Martin wrote: > Just a reminder that we are having a PGP Key Signing Party tomorrow for > WLUG. If you're name isn't listed below, I do not have your info (and > you should get it to me if you want to be on the list). I know Andy > said noon tomorrow but I'll take stuff until 4pm. Don't forget your two > forms of ID and I'll see you tomorrow. > > Eric Martin > Gregory M. Avedissian > Walt Sawyer > Alexander Pennace > Doug Mildram > Keith Wright > Alex Weiss > Andrew M. Stewart > James Philip Gray > > -Eric HI Eric, Thanks for sending out the meeting reminder! I'll see you all on Wednesday evening with my shiny new passport. :-) Later, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group (http://www.wlug.org) Chelmsford Linux Meetup Group (http://linux.meetup.com/393) Amateur Radio: KB1OIQ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGSkvGHl0iXDssISsRApWmAJ9HpIERvKHryJXnIeBo8ZfoJnqBygCeIF2I tQxc9xAb6JDHR8dbSTc3yNA= =v7Xx -----END PGP SIGNATURE-----

May 16, 2007

PGP Key Signing Party

by Eric Martin

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Just a reminder that we are having a PGP Key Signing Party tomorrow for WLUG. If you're name isn't listed below, I do not have your info (and you should get it to me if you want to be on the list). I know Andy said noon tomorrow but I'll take stuff until 4pm. Don't forget your two forms of ID and I'll see you tomorrow. Eric Martin Gregory M. Avedissian Walt Sawyer Alexander Pennace Doug Mildram Keith Wright Alex Weiss Andrew M. Stewart James Philip Gray - -Eric -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGSj+vaiVxdKlBO58RAmxgAJ0QaQlNCSrw58e80VeQjjGL/GKsOwCdErcn vMjPw9eiQD4zXEzjNWPBKa4= =KiW+ -----END PGP SIGNATURE-----

May 15, 2007

Re: [Wlug] Reminder: Upcoming PGP Key Signing Party

by Chuck Anderson

On Thu, May 10, 2007 at 08:33:31AM -0400, Mike Perry wrote: > Just an FYI. If I go to "wlug.org" I see April being the next meeting. > However, if I go to "www.wlug.org" I see the upcoming meeting. Clear your browser cache and reload the page.

May 10, 2007

Re: [Wlug] Reminder: Upcoming PGP Key Signing Party

by Mike Perry

Just an FYI. If I go to "wlug.org" I see April being the next meeting. However, if I go to "www.wlug.org" I see the upcoming meeting. ~Mike On Wed, 2007-05-09 at 20:35 -0400, Andy Stewart wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > HI Everybody, > > At our next WLUG meeting on Wednesday, May 16th, we will have a PGP Key > Signing Party hosted by Chuck Anderson and Eric Martin. Eric has > previously sent out instructions, so please be sure to follow them, and > direct any questions to Eric or Chuck. I have posted those instructions > on the WLUG home page (see description for May WLUG Meeting). > > I suggest a cutoff of noon on Wednesday, May 16th, for sending your > information to Eric. This will give Eric time to collect the necessary > information prior to the meeting. > > Dust off those passports - you'll need them! > > Later, > > Andy > > - -- > Andy Stewart, Founder > Worcester Linux Users' Group (http://www.wlug.org) > Chelmsford Linux Meetup Group (http://linux.meetup.com/393) > Amateur Radio: KB1OIQ > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.2 (GNU/Linux) > Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org > > iD8DBQFGQmjGHl0iXDssISsRAr1rAJ91nX/DNXHjRie+dDYyVjrq8ePq2wCeN8DC > Q+SoklLvcW6MfOaCGuvwKwU= > =kUPV > -----END PGP SIGNATURE----- > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug

May 10, 2007

Reminder: Upcoming PGP Key Signing Party

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 HI Everybody, At our next WLUG meeting on Wednesday, May 16th, we will have a PGP Key Signing Party hosted by Chuck Anderson and Eric Martin. Eric has previously sent out instructions, so please be sure to follow them, and direct any questions to Eric or Chuck. I have posted those instructions on the WLUG home page (see description for May WLUG Meeting). I suggest a cutoff of noon on Wednesday, May 16th, for sending your information to Eric. This will give Eric time to collect the necessary information prior to the meeting. Dust off those passports - you'll need them! Later, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group (http://www.wlug.org) Chelmsford Linux Meetup Group (http://linux.meetup.com/393) Amateur Radio: KB1OIQ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGQmjGHl0iXDssISsRAr1rAJ91nX/DNXHjRie+dDYyVjrq8ePq2wCeN8DC Q+SoklLvcW6MfOaCGuvwKwU= =kUPV -----END PGP SIGNATURE-----

May 10, 2007