- WLUG - wlug.mailman3.com

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 John Stoffel wrote: > Andy, > > The best thing to do here is to run a serial console to another one of > your systems and use THAT to try and catch the issue. You can then do > a SysRq from the remote serial console and/or catch any error > messages. > > Also, are there any messages in dmesg or /var/log/messages? How does > the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see > if the problem repeats. > > John HI John, I'm going to give the serial console a try - I may have a cable lying around. I upgraded to the 2.6.11.11 kernel which did have several fixes for K8. We'll see how that goes. Thanks for the ideas! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCp2yhHl0iXDssISsRAqxcAJwLntOh9GpNmefW/+JdwbAUMRJKBACfbV1P s2eqMUnY6rjZaIfI6MzZLqQ= =aLfl -----END PGP SIGNATURE-----

June 8, 2005

Job opportunity in Natick for experienced Unix admin

by Simoncini, Matthew

List, I work for a large consulting company in Natick, MA. and we're looking for a couple of experienced (mid level) Unix administrators to join our team. Some of the skills that are required include: Unix experience (Sun Solaris 8 or higher) Veritas volume manager Veritas cluster services Storage management experience (preferably EMC Clarion or DMX) Backup management experience (preferably EMC EDM) Team player Desire to learn and grow If you or anyone you know is looking for a position as a Unix Administrator, please contact me off list at matthewsmbox-resumes(a)yahoo.com <mailto:matthewsmbox-resumes@yahoo.com> . Thanks. Matthew

June 8, 2005

MySQL curses management tool?

by John Stoffel

Hi all, A quick googling didn't help, so I'm turning to you all. I'm looking for a mysql management tool which is CLI/curses based and more powerful/easier to use when compared to the mysqlaccess and mysqladmin stuff. Basically, something to help me manage DBs, users and permissions over an SSH connection. I know I could use some of the X tools, but I'd prefer not too if possible. Thanks, John

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by John Stoffel

Andy, The best thing to do here is to run a serial console to another one of your systems and use THAT to try and catch the issue. You can then do a SysRq from the remote serial console and/or catch any error messages. Also, are there any messages in dmesg or /var/log/messages? How does the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see if the problem repeats. John

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Jeff Moyer

==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 andystewart> Jeff Moyer wrote: >> ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart >> <andystewart(a)comcast.net> adds: >> andystewart> HI gang, >> andystewart> My dual Opteron machine is not happy. I cannot get more than andystewart> 7 straight days of uptime without getting a hard lock, andystewart> requiring a reboot. (My definition of hard lock is: machine andystewart> responds neither to keyboard input, mouse input, nor network andystewart> pings). >> andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had andystewart> 3 tonight, and 3-4 on a previous occasion while running andystewart> OpenOffice). It makes no sense to me that an application run andystewart> as a normal user could lockup a machine. >> andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an andystewart> "oops" when it hard locks - no dice. Do you know any other andystewart> tricks I could try to see if it is the kernel which is locking andystewart> up? I'm running SuSE's version of 2.6.8. >> Did you verify that NMIs are being delivered? After boot, cat >> /proc/interrupts and make sure the NMI line is non-zero. Also note >> that, at least with upstream and Red Hat kernels, the nmi_watchdog >> defaults to 1 for Opterons (i.e. you shouldn't need to manually set it). andystewart> HI Jeff, andystewart> Well, this is weird, I'm seeing a ZERO count for NMIs, so that andystewart> makes me think they are NOT being delivered. How would I go andystewart> about solving *that* little problem? I did a "cat Well, you can try booting with nmi_watchdog=2. This will try to use the local APIC to deliver nmi's, but I haven't actually seen a dual processor system that required this (all of them I've seen work with nmi_watchdog=1). It is worth a try, however. andystewart> /proc/cmdline" to insure that I had "nmi_watchdog=1" and andystewart> indeed it is there. Perhaps this is the clue we've been andystewart> seeking. Well, it only tells you that the nmi_watchdog won't trigger. We still have no insight into what the problem might actually be. >> If the NMI watchdog works, it will print a message to the console. >> However, you will not see this if you are in X windows. Do you have a >> serial console hooked up, by any chance? I strongly suggest it if you >> have the means. andystewart> I think I have a cable to which I could connect the serial andystewart> port of the Opteron to the serial port of another Linux box andystewart> (and then use minicom or some such terminal program). >> Aside from this, if it is indeed a hard lockup, there is really nothing >> you can do (without purchasing other hardware to help debug the >> problem). andystewart> Yeah, I was afraid of that. >> Please give these suggestions a shot and let us know how it goes. andystewart> Shall do - thanks, everybody! andystewart> Oh, be advised that when you smash your fist on the keyboard andystewart> after your system locks up for the umpteenth time, that a lot andystewart> of dead skin cells will come flying upward out of the bowels andystewart> of the keyboard. I would recommend safety glasses. Noted. =) -Jeff

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Jeff Moyer

==> Regarding Re: [Wlug] HOWTO debug hard lockups; Walt Sawyer <wsawyer(a)norfolk-county.com> adds: wsawyer> Andy, Are you sure it's the hard drive? Try running Knoppix live wsawyer> CD for a while. If it's still running after days(max uptime)+5, wsawyer> then perhaps it's something else. This changes things entirely. He'll be running a different kernel, so this test is not useful, in my opinion. -Jeff wsawyer> If this message is redundant, ignore it! Walt wsawyer> On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wrote: >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >> >> Jeff Moyer wrote: > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy >> Stewart <andystewart(a)comcast.net> adds: >> > >> > andystewart> HI gang, >> > >> > andystewart> My dual Opteron machine is not happy. I cannot get more >> than > andystewart> 7 straight days of uptime without getting a hard >> lock, > andystewart> requiring a reboot. (My definition of hard lock >> is: machine > andystewart> responds neither to keyboard input, mouse >> input, nor network > andystewart> pings). >> > >> > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I >> had > andystewart> 3 tonight, and 3-4 on a previous occasion while >> running > andystewart> OpenOffice). It makes no sense to me that an >> application run > andystewart> as a normal user could lockup a machine. >> > >> > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get >> an > andystewart> "oops" when it hard locks - no dice. Do you know any >> other > andystewart> tricks I could try to see if it is the kernel which >> is locking > andystewart> up? I'm running SuSE's version of 2.6.8. >> > >> > Did you verify that NMIs are being delivered? After boot, cat > >> /proc/interrupts and make sure the NMI line is non-zero. Also note >> that, > at least with upstream and Red Hat kernels, the nmi_watchdog >> defaults to 1 > for Opterons (i.e. you shouldn't need to manually set >> it). >> >> HI Jeff, >> >> Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me >> think they are NOT being delivered. How would I go about solving *that* >> little problem? I did a "cat /proc/cmdline" to insure that I had >> "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've >> been seeking. >> >> > If the NMI watchdog works, it will print a message to the console. > >> However, you will not see this if you are in X windows. Do you have a > >> serial console hooked up, by any chance? I strongly suggest it if you >> have > the means. >> >> I think I have a cable to which I could connect the serial port of the >> Opteron to the serial port of another Linux box (and then use minicom or >> some such terminal program). >> >> > >> > Aside from this, if it is indeed a hard lockup, there is really >> nothing you > can do (without purchasing other hardware to help debug >> the problem). >> >> Yeah, I was afraid of that. >> >> > >> > Please give these suggestions a shot and let us know how it goes. >> >> Shall do - thanks, everybody! >> >> Oh, be advised that when you smash your fist on the keyboard after your >> system locks up for the umpteenth time, that a lot of dead skin cells >> will come flying upward out of the bowels of the keyboard. I would >> recommend safety glasses. >> >> Later, >> >> Andy >> >> - -- >> Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA >> http://www.wlug.org >> >> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: >> Using GnuPG with Thunderbird - http://enigmail.mozdev.org >> >> iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV >> u6O8n+PSW4M0I1MHXsH06Xo= =ViVg -----END PGP SIGNATURE----- >> _______________________________________________ Wlug mailing list >> Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug >> wsawyer> -- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN"> wsawyer> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; wsawyer> CHARSET=UTF-8"> <META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2"> wsawyer> </HEAD> <BODY> Andy, Are you sure it's the hard drive?  wsawyer> Try running Knoppix live CD for a while.  If it's still wsawyer> running after days(max uptime)+5, then perhaps it's something wsawyer> else. If this message is redundant, ignore it! wsawyer> Walt On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wsawyer> wrote: <BLOCKQUOTE TYPE=CITE> <PRE> COLOR="#000000">-----BEGIN PGP SIGNED MESSAGE----- COLOR="#000000">Hash: SHA1 wsawyer> Jeff Moyer wrote: COLOR="#000000">> ==> Regarding [Wlug] HOWTO debug hard wsawyer> lockups; Andy Stewart <<A wsawyer> HREF="mailto:andystewart@comcast.net">andystewart(a)comcast.net</A>> wsawyer> adds: > COLOR="#000000">> andystewart> HI gang, COLOR="#000000">> > wsawyer> andystewart> My dual Opteron machine is not happy. I cannot wsawyer> get more than > andystewart> 7 wsawyer> straight days of uptime without getting a hard lock, COLOR="#000000">> andystewart> requiring a reboot. (My wsawyer> definition of hard lock is: machine COLOR="#000000">> andystewart> responds neither to keyboard wsawyer> input, mouse input, nor network > wsawyer> andystewart> pings). > wsawyer> > andystewart> I can stimulate hard wsawyer> locks by running OpenOffice 1.1.3 (I had COLOR="#000000">> andystewart> 3 tonight, and 3-4 on a wsawyer> previous occasion while running > wsawyer> andystewart> OpenOffice). It makes no sense to me that an wsawyer> application run > andystewart> wsawyer> as a normal user could lockup a machine. COLOR="#000000">> > wsawyer> andystewart> I've tried setting "nmi_watchdog=1" to wsawyer> see if I could get an > wsawyer> andystewart> "oops" when it hard locks - no dice. Do wsawyer> you know any other > wsawyer> andystewart> tricks I could try to see if it is the kernel wsawyer> which is locking > andystewart> wsawyer> up? I'm running SuSE's version of 2.6.8. COLOR="#000000">> > Did you wsawyer> verify that NMIs are being delivered? After boot, cat wsawyer> > /proc/interrupts and make sure the NMI wsawyer> line is non-zero. Also note that, COLOR="#000000">> at least with upstream and Red Hat kernels, wsawyer> the nmi_watchdog defaults to 1 > wsawyer> for Opterons (i.e. you shouldn't need to manually set it). wsawyer> HI Jeff, wsawyer> Well, this is weird, I'm seeing a ZERO count wsawyer> for NMIs, so that makes me think they wsawyer> are NOT being delivered. How would I go about solving wsawyer> *that* little problem? I did a wsawyer> "cat /proc/cmdline" to insure that I had COLOR="#000000">"nmi_watchdog=1" and indeed it is there. wsawyer> Perhaps this is the clue we've been wsawyer> seeking. wsawyer> > If the NMI watchdog works, it will wsawyer> print a message to the console. > wsawyer> However, you will not see this if you are in X windows. Do you wsawyer> have a > serial console hooked up, wsawyer> by any chance? I strongly suggest it if you have COLOR="#000000">> the means. wsawyer> I think I have a cable to which I could wsawyer> connect the serial port of the COLOR="#000000">Opteron to the serial port of another Linux box wsawyer> (and then use minicom or some such wsawyer> terminal program). wsawyer> > > wsawyer> Aside from this, if it is indeed a hard lockup, there is really wsawyer> nothing you > can do (without wsawyer> purchasing other hardware to help debug the problem). wsawyer> Yeah, I was afraid of that. wsawyer> > > wsawyer> Please give these suggestions a shot and let us know how it wsawyer> goes. wsawyer> Shall do - thanks, everybody! wsawyer> Oh, be advised that when you smash your fist wsawyer> on the keyboard after your system wsawyer> locks up for the umpteenth time, that a lot of dead skin wsawyer> cells will come flying upward out of wsawyer> the bowels of the keyboard. I would COLOR="#000000">recommend safety glasses. wsawyer> Later, wsawyer> Andy wsawyer> - -- Andy wsawyer> Stewart, Founder Worcester Linux wsawyer> Users' Group Worcester, MA, wsawyer> USA <A wsawyer> HREF="http://www.wlug.org">http://www.wlug.org</A> wsawyer> -----BEGIN PGP SIGNATURE----- COLOR="#000000">Version: GnuPG v1.2.5 (GNU/Linux) COLOR="#000000">Comment: Using GnuPG with Thunderbird - <A wsawyer> HREF="http://enigmail.mozdev.org">http://enigmail.mozdev.org</A> wsawyer> COLOR="#000000">iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV wsawyer> u6O8n+PSW4M0I1MHXsH06Xo= COLOR="#000000">=ViVg -----END PGP wsawyer> SIGNATURE----- COLOR="#000000">_______________________________________________ wsawyer> Wlug mailing list COLOR="#000000"><A wsawyer> HREF="mailto:Wlug@mail.wlug.org">Wlug(a)mail.wlug.org</A> wsawyer> <A wsawyer> HREF="http://mail.wlug.org/mailman/listinfo/wlug">http://mail.wlug.org/mailman/listinfo/wlug</A> wsawyer> </PRE> </BLOCKQUOTE> <TABLE CELLSPACING="0" CELLPADDING="0" wsawyer> WIDTH="100%"> <TR> <TD> <PRE> -- </PRE> </TD> </TR> </TABLE> wsawyer> </BODY> </HTML> _______________________________________________ wsawyer> Wlug mailing list Wlug(a)mail.wlug.org wsawyer> http://mail.wlug.org/mailman/listinfo/wlug

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Gary Hanley

On Tue, 7 Jun 2005, Andy Stewart wrote: > I'd rather not give up KDE, but I could try blackbox for a while to see > what happens. Well, my guess is that it is a hardware issue How about the MB settings, like the memory timings/clock ratios? Sometimes this can cause problems if they are set too aggressively. Usually there are one-off options in the BIOS menu for stable settings and speed settings so you don't have to tweak everything. Anything there you can check? Maybe down-shift your settings a little and see if the problem goes away. I haven't had the pleasure of working with a dual system every so I'm not familiar with any related BIOS settings you may need to modify. -- Gary

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Baris Hasdemir wrote: > On Tue, 2005-06-07 at 20:10 -0400, Andy Stewart wrote: >>-----BEGIN PGP SIGNED MESSAGE----- >>Hash: SHA1 >> >>Baris Hasdemir wrote: >>>On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: >>>>I'd rather not give up KDE, but I could try blackbox for a while to see >>>>what happens. >>>It might also worth to disable hardware acceleration support. For >>>example you can use nv instead of nvidia, for a while. >>> >>>baris >>HI Baris, >> >>Thanks for the tip, but my video card is an older one based on the ATI >>Radeon 7500. Not everything in this machine is new. >> >>Thanks, > > Are you using ati or radeon module? if using latter you have the > hardware acceleration support, if I recall correctly. > HI Baris, Hmmm, let me check. It appears that I'm using the radeon module. I'll make a note to try disabling hardware acceleration support for the video card. Thanks for the idea! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpl32Hl0iXDssISsRAuKKAJ9nkqIT8By9WV4h9htjaiy7Sb/KpACbBmwF jInvBZ/LKPzIlezoXl7Bc+k= =qs7r -----END PGP SIGNATURE-----

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Baris Hasdemir

On Tue, 2005-06-07 at 20:10 -0400, Andy Stewart wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Baris Hasdemir wrote: > > On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: > >>I'd rather not give up KDE, but I could try blackbox for a while to see > >>what happens. > > > > It might also worth to disable hardware acceleration support. For > > example you can use nv instead of nvidia, for a while. > > > > baris > > HI Baris, > > Thanks for the tip, but my video card is an older one based on the ATI > Radeon 7500. Not everything in this machine is new. > > Thanks, Are you using ati or radeon module? if using latter you have the hardware acceleration support, if I recall correctly. baris

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Walt Sawyer wrote: > Andy, > Are you sure it's the hard drive? Try running Knoppix live CD for a > while. If it's still running after days(max uptime)+5, then perhaps > it's something else. > > If this message is redundant, ignore it! > Walt HI Walt, I have suffered from hard drive problems on previous systems, and in each of those cases, I had strange errors reported in the /var/log/messages file. That's not to say that a hard drive problem couldn't cause a lockup, but that hasn't been my past experience. With the lockups I'm currently seeing, I'm not seeing any error messages in the /var/log/messages file. At this point, I think anything and everything is suspect until proven otherwise. Thanks for the ideas! I'm jotting them all down. Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpjjYHl0iXDssISsRAvQFAJ4w8Q4A5Ai25Se0pgkoKdYWo9AFcACdF3Q4 bC9BhLL4vESujGOMSsA4R2o= =+CpT -----END PGP SIGNATURE-----

June 8, 2005