- WLUG - wlug.mailman3.com

large file performance on linux
by brad noyes May 16, 2007

May 16, 2007

Hello All, I am seeing some really slow performance regarding large files on linux. I write a lot of data points from a light sensor. The stream is about 53 Mb/s and i need to keep this rate for 7 minutes, that's a total of about 22Gb. I can sustain 53Mb/s pretty well until the file grows to over 1Gb or so, then things hit the wall and the writes to the filesystem can't keep up. The writes go from 20ms in duration to 500ms. I assume the filesystem/operating system is caching writes. Do you have any suggestions on how to speed up performance on these writes, filesystem options, kernel options, other strategies, etc? Things I have tried: - I have tried this on a ext3 file system as well as an xfs filesystem with the same result. - I have also tried spooling over several files (a la multiple volumes) but i see no difference in performance. In fact, i think this actually hinders performance a bit. - I keep my own giant memory buffer where all the data is stored and then it is written to disk in a background thread. This helps, but i run out of space in the buffer before i finish taking data. Thanks, -- Brad

6 13

Meeting tomight
by freak4uxxx＠gmail.com May 16, 2007

May 16, 2007

I might be slightly delayed tonight, small family thing that needs to be taken care of. I shouldn't be late but I just wanted to let people know. -Eric

2 1

Reminder: Upcoming PGP Key Signing Party
by Andy Stewart May 16, 2007

May 16, 2007

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 HI Everybody, At our next WLUG meeting on Wednesday, May 16th, we will have a PGP Key Signing Party hosted by Chuck Anderson and Eric Martin. Eric has previously sent out instructions, so please be sure to follow them, and direct any questions to Eric or Chuck. I have posted those instructions on the WLUG home page (see description for May WLUG Meeting). I suggest a cutoff of noon on Wednesday, May 16th, for sending your information to Eric. This will give Eric time to collect the necessary information prior to the meeting. Dust off those passports - you'll need them! Later, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group (http://www.wlug.org) Chelmsford Linux Meetup Group (http://linux.meetup.com/393) Amateur Radio: KB1OIQ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGQmjGHl0iXDssISsRAr1rAJ91nX/DNXHjRie+dDYyVjrq8ePq2wCeN8DC Q+SoklLvcW6MfOaCGuvwKwU= =kUPV -----END PGP SIGNATURE-----

4 3

Re: [Wlug] large file performance on linux
by Jamie Guinan May 16, 2007

May 16, 2007

On Wed, 16 May 2007, brad noyes wrote: > Hello All, > > I am seeing some really slow performance regarding large files on linux. I > write a lot of data points from a light sensor. The stream is about 53 Mb/s and > i need to keep this rate for 7 minutes, that's a total of about 22Gb. I > can sustain 53Mb/s pretty well until the file grows to over 1Gb or so, then > things hit the wall and the writes to the filesystem can't keep up. The writes > go from 20ms in duration to 500ms. I assume the filesystem/operating system > is caching writes. Do you have any suggestions on how to speed up performance > on these writes, filesystem options, kernel options, other strategies, etc? > > Things I have tried: > - I have tried this on a ext3 file system as well as an xfs filesystem > with the same result. > > - I have also tried spooling over several files (a la multiple volumes) > but i see no difference in performance. In fact, i think this actually > hinders performance a bit. > > - I keep my own giant memory buffer where all the data is stored and > then it is written to disk in a background thread. This helps, but > i run out of space in the buffer before i finish taking data. Hi, Do you mean MB/s as in MegaBytes/second? Lower-case "b" implies "bits" to me, but I suspect you meant bytes. :) Anyway, if you know your total data set is going to exceed your system memory, which will largely get used as page cache, you might as well open the output with O_SYNC and write straight out. Then the big buffer in the writer thread can go away, and you can just queue chunks between the input and writer. A good experiment would be to run the writer thread with dummy input data and see what kind of throughput you get. You could try ext2 (no journalling), or tweaking some of the journalling options in ext3 (data=journal/ordered/writeback, see "man mount"). If you have enough CPU bandwidth, and your input stream has enough redundancy, you could gzip it before writing, which might reduce your output bandwidth requirements. Hope this helps, keep us posted. -Jamie > > Thanks, > -- Brad > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug >

2 1

SUN Ultra1 booting (still, sorry)
by ken jones May 8, 2007

May 8, 2007

I used genkernel to compile my 2.6.20-gentoo-r5 kernel. After much thrashing I got it to boot. All looked good until the very end when it claimed that it "Failed to set the user font" announced the name of my machine and gave me the login prompt. It would not respond to any keyboard input. After a minute or two it printed a line about "disabled for 5 minutes" and went permanently dead. See below. Any ideas? ---------------snip from almost successful boot--------------- * Starting lo [ !! * Bringing up lo * 127.0.0.1/8 * Adding routes [ ok * 127.0.0.0/8 ... * Initializing random number generator ... [ ok NIT: Entering runlevel: 3 [ ok * Setting user font ... putfont: PIO_FONT: Function not implemented putfont: PIO_FONT: Function not implemented putfont: PIO_FONT: Function not implemented putfont: PIO_FONT: Function not implemented putfont: PIO_FONT: Function not implemented putfont: PIO_FONT: Function not implemented putfont: PIO_FONT: Function not implemented putfont: PIO_FONT: Function not implemented putfont: PIO_FONT: Function not implemented putfont: PIO_FONT: Function not implemented putfont: PIO_FONT: Function not implemented * Failed to set user font * Mounting network filesystems ... [ !! * Starting local ... [ ok [ ok This is kjones-sun2.unknown_domain (Linux sparc64 2.6.20-gentoo-r5) 21:42:42 kjones-sun2 login: INIT: Id "s0" respawning too fast: disabled for 5 minutes

1 0

[OT] ethics
by Mark Richards May 2, 2007

May 2, 2007

Hi Matt. Matt Shields wrote: > I haven't checked out CPSR, so I don't know what they are, but you > can't enforce your beliefs on others. This comes down to personal > choice and where you want to work, or maybe working for the government > is the only available choice at the time. Some people have no problem I agree. What I was suggesting is the possibility for one or more people to make a statement as to where they choose to allow the fruit of their work to go, and/or how it is to be used. That's not about enforcing beliefs, but about sharing (and refining) them. The problem I have with open source licensing as it applies to this discussion I raised is that it does not discriminate as to use. I had once posed the possibility of creating a modified license that would prohibit (any) government from using a certain work, but this is both way too complicated and of course begs the question as to exactly what in a project can be legally restricted in this manner. After all, any project (including the one I am working on now) is comprised of a host of others, all bringing in license terms that do not have such a restriction. Attorneys will be needed to figure out if the idea of a more restrictive version of a GPL is even possible. > working for the government. If you want to change how the government > works because you don't like what they're doing, you elect officials > to change policy and petition those officials to change things. You Well, I think that the effectiveness of the system itself is a whole other debate that would go way off topic here. :) I do find the idea of a national referendum on important issues like blowing up other nations a pretty good idea. Has immediacy. But again, swaying way off here. > don't come up with a society that boycotts working for the government > because there's plenty of other people willing to take the job. There are plenty of people. But, as in organizations like Union of Concerned Scientists (the "concerned", I think, puts the intensity of the overall problem mildly), people have come together and made quite a difference. If they hadn't tried then the issues they raise may well have remained in obscurity while scientists of conscience masked their understanding and "concern" with lots of rolaids. /m

1 0

Re: [Wlug] JOBS: Seeking a summer intern in Linux administration
by Clint Moyer May 2, 2007

May 2, 2007

At least that's only the job listing. If you _apply_ for a unix job with a MS Word resume, you get auto-deleted. ---- Jared Greenwald <greenwaldjared(a)gmail.com> wrote: ============= Nothing says Linux like a Word doc... On 5/1/07, Andy Stewart <andystewart(a)comcast.net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > HI Everybody, > > If you have an interest in this position, please reply directly to the > recruiter: > > Mischke, Ali <Ali.Mischke(a)ironmountain.com> > > Please note: when I forward email such as this, I'll be putting "JOBS:" > or "JOB:" at the beginning of the subject. You may thus filter at will. > > Thanks, > > Andy > > === > > Does WLUG have a student forum? I am the Engineering Internship > Coordinator at Iron Mountain Digital in Southborough, MA, and am trying > to get the word out about a paid internship we have for this summer. We > are looking for a student (junior, senior or graduate student) with > strong Linux skills to help administer an Engineering lab of 125-175 > Linux servers. I have attached the job description and would appreciate > if you could help get the word out to any student members of your > organization. > > Thanks in advance! > > Ali Mischke > Senior Manager, Analysis and Design > Iron Mountain Digital > > 120 Turnpike Road > Southborough, MA 01772 > Phone: (508) 808-7347 > ali.mischke(a)ironmountain.com > > === > > > - -- > Andy Stewart, Founder > Worcester Linux Users' Group (http://www.wlug.org) > Chelmsford Linux Meetup Group (http://linux.meetup.com/393) > Amateur Radio: KB1OIQ > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.2 (GNU/Linux) > Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org > > iD8DBQFGN7l3Hl0iXDssISsRAiGVAJ9PcqnYps7gc+2SZLc9cfXAz6mhWwCcCW4Y > 7c062GdyqG5Rw5nDEwT009I= > =0Jq9 > -----END PGP SIGNATURE----- > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > > > _______________________________________________ Wlug mailing list Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug

2 1

JOBS: Seeking a summer intern in Linux administration
by Andy Stewart May 2, 2007

May 2, 2007

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 HI Everybody, If you have an interest in this position, please reply directly to the recruiter: Mischke, Ali <Ali.Mischke(a)ironmountain.com> Please note: when I forward email such as this, I'll be putting "JOBS:" or "JOB:" at the beginning of the subject. You may thus filter at will. Thanks, Andy === Does WLUG have a student forum? I am the Engineering Internship Coordinator at Iron Mountain Digital in Southborough, MA, and am trying to get the word out about a paid internship we have for this summer. We are looking for a student (junior, senior or graduate student) with strong Linux skills to help administer an Engineering lab of 125-175 Linux servers. I have attached the job description and would appreciate if you could help get the word out to any student members of your organization. Thanks in advance! Ali Mischke Senior Manager, Analysis and Design Iron Mountain Digital 120 Turnpike Road Southborough, MA 01772 Phone: (508) 808-7347 ali.mischke(a)ironmountain.com === - -- Andy Stewart, Founder Worcester Linux Users' Group (http://www.wlug.org) Chelmsford Linux Meetup Group (http://linux.meetup.com/393) Amateur Radio: KB1OIQ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGN7l3Hl0iXDssISsRAiGVAJ9PcqnYps7gc+2SZLc9cfXAz6mhWwCcCW4Y 7c062GdyqG5Rw5nDEwT009I= =0Jq9 -----END PGP SIGNATURE-----

4 4

MySQL Server Hardware
by John Westcott, IV April 30, 2007

April 30, 2007

Howdy All, I work in a place that has traditionally used Sun Sparc hardware running Solaris to host a production MySQL environment. We are going to be replacing our database servers this year and I am trying to determine if I want to maintain the norm or if I want to try to change. I am interested to see what other people use for hardware/OS for a production MySQL installation that requires a high amount of uptime. Thanks in advance for any replies. -John Westcott

6 5

I'd like to see Windows do this....
by Tal Cohen April 30, 2007

April 30, 2007

Hi Folks, I recently ran across an interesting situation that I wanted to share this group. A customer of ours called us, letting us know that our server in their data center was beeping. We tried to ssh into the box, but kept getting connection refused. I asked the customer to hook up a monitor and keyboard to the box, and try a simple TTY. Sure enough, he was unable to log in. It turns out that the hard drive that holds the OS (Crux Linux) failed. Now, that in and of itself is not very interesting. What is interesting is that our software kept running even without the OS drive functioning. Our software resides on a separate drive. Since it was already loaded into memory and almost never needs to touch the OS drive, it just kept going.and going..and going.. Tal

3 5