- WLUG - wlug.mailman3.com

Re: [Wlug] HOWTO debug hard lockups

by Michael Long

Andy Stewart wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >John Stoffel wrote: > > >>Andy, >> >>The best thing to do here is to run a serial console to another one of >>your systems and use THAT to try and catch the issue. You can then do >>a SysRq from the remote serial console and/or catch any error >>messages. >> >>Also, are there any messages in dmesg or /var/log/messages? How does >>the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see >>if the problem repeats. >> >>John >> >> > >HI John, > >I'm going to give the serial console a try - I may have a cable lying >around. > >I upgraded to the 2.6.11.11 kernel which did have several fixes for K8. > We'll see how that goes. > >Thanks for the ideas! > >Andy > >- -- >Andy Stewart, Founder >Worcester Linux Users' Group >Worcester, MA, USA >http://www.wlug.org > >-----BEGIN PGP SIGNATURE----- >Version: GnuPG v1.2.5 (GNU/Linux) >Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > >iD8DBQFCp2yhHl0iXDssISsRAqxcAJwLntOh9GpNmefW/+JdwbAUMRJKBACfbV1P >s2eqMUnY6rjZaIfI6MzZLqQ= >=aLfl >-----END PGP SIGNATURE----- >_______________________________________________ >Wlug mailing list >Wlug(a)mail.wlug.org >http://mail.wlug.org/mailman/listinfo/wlug > > Andy, If that doesn't help try using the system without the kvm and see if you get the same behavior. I have seen odd things in the past when using a faulty kvm switch. Mike

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 John Stoffel wrote: > Andy, > > The best thing to do here is to run a serial console to another one of > your systems and use THAT to try and catch the issue. You can then do > a SysRq from the remote serial console and/or catch any error > messages. > > Also, are there any messages in dmesg or /var/log/messages? How does > the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see > if the problem repeats. > > John HI John, I'm going to give the serial console a try - I may have a cable lying around. I upgraded to the 2.6.11.11 kernel which did have several fixes for K8. We'll see how that goes. Thanks for the ideas! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCp2yhHl0iXDssISsRAqxcAJwLntOh9GpNmefW/+JdwbAUMRJKBACfbV1P s2eqMUnY6rjZaIfI6MzZLqQ= =aLfl -----END PGP SIGNATURE-----

June 8, 2005

Job opportunity in Natick for experienced Unix admin

by Simoncini, Matthew

List, I work for a large consulting company in Natick, MA. and we're looking for a couple of experienced (mid level) Unix administrators to join our team. Some of the skills that are required include: Unix experience (Sun Solaris 8 or higher) Veritas volume manager Veritas cluster services Storage management experience (preferably EMC Clarion or DMX) Backup management experience (preferably EMC EDM) Team player Desire to learn and grow If you or anyone you know is looking for a position as a Unix Administrator, please contact me off list at matthewsmbox-resumes(a)yahoo.com <mailto:matthewsmbox-resumes@yahoo.com> . Thanks. Matthew

June 8, 2005

MySQL curses management tool?

by John Stoffel

Hi all, A quick googling didn't help, so I'm turning to you all. I'm looking for a mysql management tool which is CLI/curses based and more powerful/easier to use when compared to the mysqlaccess and mysqladmin stuff. Basically, something to help me manage DBs, users and permissions over an SSH connection. I know I could use some of the X tools, but I'd prefer not too if possible. Thanks, John

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by John Stoffel

Andy, The best thing to do here is to run a serial console to another one of your systems and use THAT to try and catch the issue. You can then do a SysRq from the remote serial console and/or catch any error messages. Also, are there any messages in dmesg or /var/log/messages? How does the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see if the problem repeats. John

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Jeff Moyer

==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 andystewart> Jeff Moyer wrote: >> ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart >> <andystewart(a)comcast.net> adds: >> andystewart> HI gang, >> andystewart> My dual Opteron machine is not happy. I cannot get more than andystewart> 7 straight days of uptime without getting a hard lock, andystewart> requiring a reboot. (My definition of hard lock is: machine andystewart> responds neither to keyboard input, mouse input, nor network andystewart> pings). >> andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had andystewart> 3 tonight, and 3-4 on a previous occasion while running andystewart> OpenOffice). It makes no sense to me that an application run andystewart> as a normal user could lockup a machine. >> andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an andystewart> "oops" when it hard locks - no dice. Do you know any other andystewart> tricks I could try to see if it is the kernel which is locking andystewart> up? I'm running SuSE's version of 2.6.8. >> Did you verify that NMIs are being delivered? After boot, cat >> /proc/interrupts and make sure the NMI line is non-zero. Also note >> that, at least with upstream and Red Hat kernels, the nmi_watchdog >> defaults to 1 for Opterons (i.e. you shouldn't need to manually set it). andystewart> HI Jeff, andystewart> Well, this is weird, I'm seeing a ZERO count for NMIs, so that andystewart> makes me think they are NOT being delivered. How would I go andystewart> about solving *that* little problem? I did a "cat Well, you can try booting with nmi_watchdog=2. This will try to use the local APIC to deliver nmi's, but I haven't actually seen a dual processor system that required this (all of them I've seen work with nmi_watchdog=1). It is worth a try, however. andystewart> /proc/cmdline" to insure that I had "nmi_watchdog=1" and andystewart> indeed it is there. Perhaps this is the clue we've been andystewart> seeking. Well, it only tells you that the nmi_watchdog won't trigger. We still have no insight into what the problem might actually be. >> If the NMI watchdog works, it will print a message to the console. >> However, you will not see this if you are in X windows. Do you have a >> serial console hooked up, by any chance? I strongly suggest it if you >> have the means. andystewart> I think I have a cable to which I could connect the serial andystewart> port of the Opteron to the serial port of another Linux box andystewart> (and then use minicom or some such terminal program). >> Aside from this, if it is indeed a hard lockup, there is really nothing >> you can do (without purchasing other hardware to help debug the >> problem). andystewart> Yeah, I was afraid of that. >> Please give these suggestions a shot and let us know how it goes. andystewart> Shall do - thanks, everybody! andystewart> Oh, be advised that when you smash your fist on the keyboard andystewart> after your system locks up for the umpteenth time, that a lot andystewart> of dead skin cells will come flying upward out of the bowels andystewart> of the keyboard. I would recommend safety glasses. Noted. =) -Jeff

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Jeff Moyer

==> Regarding Re: [Wlug] HOWTO debug hard lockups; Walt Sawyer <wsawyer(a)norfolk-county.com> adds: wsawyer> Andy, Are you sure it's the hard drive? Try running Knoppix live wsawyer> CD for a while. If it's still running after days(max uptime)+5, wsawyer> then perhaps it's something else. This changes things entirely. He'll be running a different kernel, so this test is not useful, in my opinion. -Jeff wsawyer> If this message is redundant, ignore it! Walt wsawyer> On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wrote: >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >> >> Jeff Moyer wrote: > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy >> Stewart <andystewart(a)comcast.net> adds: >> > >> > andystewart> HI gang, >> > >> > andystewart> My dual Opteron machine is not happy. I cannot get more >> than > andystewart> 7 straight days of uptime without getting a hard >> lock, > andystewart> requiring a reboot. (My definition of hard lock >> is: machine > andystewart> responds neither to keyboard input, mouse >> input, nor network > andystewart> pings). >> > >> > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I >> had > andystewart> 3 tonight, and 3-4 on a previous occasion while >> running > andystewart> OpenOffice). It makes no sense to me that an >> application run > andystewart> as a normal user could lockup a machine. >> > >> > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get >> an > andystewart> "oops" when it hard locks - no dice. Do you know any >> other > andystewart> tricks I could try to see if it is the kernel which >> is locking > andystewart> up? I'm running SuSE's version of 2.6.8. >> > >> > Did you verify that NMIs are being delivered? After boot, cat > >> /proc/interrupts and make sure the NMI line is non-zero. Also note >> that, > at least with upstream and Red Hat kernels, the nmi_watchdog >> defaults to 1 > for Opterons (i.e. you shouldn't need to manually set >> it). >> >> HI Jeff, >> >> Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me >> think they are NOT being delivered. How would I go about solving *that* >> little problem? I did a "cat /proc/cmdline" to insure that I had >> "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've >> been seeking. >> >> > If the NMI watchdog works, it will print a message to the console. > >> However, you will not see this if you are in X windows. Do you have a > >> serial console hooked up, by any chance? I strongly suggest it if you >> have > the means. >> >> I think I have a cable to which I could connect the serial port of the >> Opteron to the serial port of another Linux box (and then use minicom or >> some such terminal program). >> >> > >> > Aside from this, if it is indeed a hard lockup, there is really >> nothing you > can do (without purchasing other hardware to help debug >> the problem). >> >> Yeah, I was afraid of that. >> >> > >> > Please give these suggestions a shot and let us know how it goes. >> >> Shall do - thanks, everybody! >> >> Oh, be advised that when you smash your fist on the keyboard after your >> system locks up for the umpteenth time, that a lot of dead skin cells >> will come flying upward out of the bowels of the keyboard. I would >> recommend safety glasses. >> >> Later, >> >> Andy >> >> - -- >> Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA >> http://www.wlug.org >> >> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: >> Using GnuPG with Thunderbird - http://enigmail.mozdev.org >> >> iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV >> u6O8n+PSW4M0I1MHXsH06Xo= =ViVg -----END PGP SIGNATURE----- >> _______________________________________________ Wlug mailing list >> Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug >> wsawyer> -- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN"> wsawyer> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; wsawyer> CHARSET=UTF-8"> <META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2"> wsawyer> </HEAD> <BODY> Andy, Are you sure it's the hard drive?  wsawyer> Try running Knoppix live CD for a while.  If it's still wsawyer> running after days(max uptime)+5, then perhaps it's something wsawyer> else. If this message is redundant, ignore it! wsawyer> Walt On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wsawyer> wrote: <BLOCKQUOTE TYPE=CITE> <PRE> COLOR="#000000">-----BEGIN PGP SIGNED MESSAGE----- COLOR="#000000">Hash: SHA1 wsawyer> Jeff Moyer wrote: COLOR="#000000">> ==> Regarding [Wlug] HOWTO debug hard wsawyer> lockups; Andy Stewart <<A wsawyer> HREF="mailto:andystewart@comcast.net">andystewart(a)comcast.net</A>> wsawyer> adds: > COLOR="#000000">> andystewart> HI gang, COLOR="#000000">> > wsawyer> andystewart> My dual Opteron machine is not happy. I cannot wsawyer> get more than > andystewart> 7 wsawyer> straight days of uptime without getting a hard lock, COLOR="#000000">> andystewart> requiring a reboot. (My wsawyer> definition of hard lock is: machine COLOR="#000000">> andystewart> responds neither to keyboard wsawyer> input, mouse input, nor network > wsawyer> andystewart> pings). > wsawyer> > andystewart> I can stimulate hard wsawyer> locks by running OpenOffice 1.1.3 (I had COLOR="#000000">> andystewart> 3 tonight, and 3-4 on a wsawyer> previous occasion while running > wsawyer> andystewart> OpenOffice). It makes no sense to me that an wsawyer> application run > andystewart> wsawyer> as a normal user could lockup a machine. COLOR="#000000">> > wsawyer> andystewart> I've tried setting "nmi_watchdog=1" to wsawyer> see if I could get an > wsawyer> andystewart> "oops" when it hard locks - no dice. Do wsawyer> you know any other > wsawyer> andystewart> tricks I could try to see if it is the kernel wsawyer> which is locking > andystewart> wsawyer> up? I'm running SuSE's version of 2.6.8. COLOR="#000000">> > Did you wsawyer> verify that NMIs are being delivered? After boot, cat wsawyer> > /proc/interrupts and make sure the NMI wsawyer> line is non-zero. Also note that, COLOR="#000000">> at least with upstream and Red Hat kernels, wsawyer> the nmi_watchdog defaults to 1 > wsawyer> for Opterons (i.e. you shouldn't need to manually set it). wsawyer> HI Jeff, wsawyer> Well, this is weird, I'm seeing a ZERO count wsawyer> for NMIs, so that makes me think they wsawyer> are NOT being delivered. How would I go about solving wsawyer> *that* little problem? I did a wsawyer> "cat /proc/cmdline" to insure that I had COLOR="#000000">"nmi_watchdog=1" and indeed it is there. wsawyer> Perhaps this is the clue we've been wsawyer> seeking. wsawyer> > If the NMI watchdog works, it will wsawyer> print a message to the console. > wsawyer> However, you will not see this if you are in X windows. Do you wsawyer> have a > serial console hooked up, wsawyer> by any chance? I strongly suggest it if you have COLOR="#000000">> the means. wsawyer> I think I have a cable to which I could wsawyer> connect the serial port of the COLOR="#000000">Opteron to the serial port of another Linux box wsawyer> (and then use minicom or some such wsawyer> terminal program). wsawyer> > > wsawyer> Aside from this, if it is indeed a hard lockup, there is really wsawyer> nothing you > can do (without wsawyer> purchasing other hardware to help debug the problem). wsawyer> Yeah, I was afraid of that. wsawyer> > > wsawyer> Please give these suggestions a shot and let us know how it wsawyer> goes. wsawyer> Shall do - thanks, everybody! wsawyer> Oh, be advised that when you smash your fist wsawyer> on the keyboard after your system wsawyer> locks up for the umpteenth time, that a lot of dead skin wsawyer> cells will come flying upward out of wsawyer> the bowels of the keyboard. I would COLOR="#000000">recommend safety glasses. wsawyer> Later, wsawyer> Andy wsawyer> - -- Andy wsawyer> Stewart, Founder Worcester Linux wsawyer> Users' Group Worcester, MA, wsawyer> USA <A wsawyer> HREF="http://www.wlug.org">http://www.wlug.org</A> wsawyer> -----BEGIN PGP SIGNATURE----- COLOR="#000000">Version: GnuPG v1.2.5 (GNU/Linux) COLOR="#000000">Comment: Using GnuPG with Thunderbird - <A wsawyer> HREF="http://enigmail.mozdev.org">http://enigmail.mozdev.org</A> wsawyer> COLOR="#000000">iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV wsawyer> u6O8n+PSW4M0I1MHXsH06Xo= COLOR="#000000">=ViVg -----END PGP wsawyer> SIGNATURE----- COLOR="#000000">_______________________________________________ wsawyer> Wlug mailing list COLOR="#000000"><A wsawyer> HREF="mailto:Wlug@mail.wlug.org">Wlug(a)mail.wlug.org</A> wsawyer> <A wsawyer> HREF="http://mail.wlug.org/mailman/listinfo/wlug">http://mail.wlug.org/mailman/listinfo/wlug</A> wsawyer> </PRE> </BLOCKQUOTE> <TABLE CELLSPACING="0" CELLPADDING="0" wsawyer> WIDTH="100%"> <TR> <TD> <PRE> -- </PRE> </TD> </TR> </TABLE> wsawyer> </BODY> </HTML> _______________________________________________ wsawyer> Wlug mailing list Wlug(a)mail.wlug.org wsawyer> http://mail.wlug.org/mailman/listinfo/wlug

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Gary Hanley

On Tue, 7 Jun 2005, Andy Stewart wrote: > I'd rather not give up KDE, but I could try blackbox for a while to see > what happens. Well, my guess is that it is a hardware issue How about the MB settings, like the memory timings/clock ratios? Sometimes this can cause problems if they are set too aggressively. Usually there are one-off options in the BIOS menu for stable settings and speed settings so you don't have to tweak everything. Anything there you can check? Maybe down-shift your settings a little and see if the problem goes away. I haven't had the pleasure of working with a dual system every so I'm not familiar with any related BIOS settings you may need to modify. -- Gary

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Baris Hasdemir wrote: > On Tue, 2005-06-07 at 20:10 -0400, Andy Stewart wrote: >>-----BEGIN PGP SIGNED MESSAGE----- >>Hash: SHA1 >> >>Baris Hasdemir wrote: >>>On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: >>>>I'd rather not give up KDE, but I could try blackbox for a while to see >>>>what happens. >>>It might also worth to disable hardware acceleration support. For >>>example you can use nv instead of nvidia, for a while. >>> >>>baris >>HI Baris, >> >>Thanks for the tip, but my video card is an older one based on the ATI >>Radeon 7500. Not everything in this machine is new. >> >>Thanks, > > Are you using ati or radeon module? if using latter you have the > hardware acceleration support, if I recall correctly. > HI Baris, Hmmm, let me check. It appears that I'm using the radeon module. I'll make a note to try disabling hardware acceleration support for the video card. Thanks for the idea! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpl32Hl0iXDssISsRAuKKAJ9nkqIT8By9WV4h9htjaiy7Sb/KpACbBmwF jInvBZ/LKPzIlezoXl7Bc+k= =qs7r -----END PGP SIGNATURE-----

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Baris Hasdemir

On Tue, 2005-06-07 at 20:10 -0400, Andy Stewart wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Baris Hasdemir wrote: > > On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: > >>I'd rather not give up KDE, but I could try blackbox for a while to see > >>what happens. > > > > It might also worth to disable hardware acceleration support. For > > example you can use nv instead of nvidia, for a while. > > > > baris > > HI Baris, > > Thanks for the tip, but my video card is an older one based on the ATI > Radeon 7500. Not everything in this machine is new. > > Thanks, Are you using ati or radeon module? if using latter you have the hardware acceleration support, if I recall correctly. baris

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Walt Sawyer wrote: > Andy, > Are you sure it's the hard drive? Try running Knoppix live CD for a > while. If it's still running after days(max uptime)+5, then perhaps > it's something else. > > If this message is redundant, ignore it! > Walt HI Walt, I have suffered from hard drive problems on previous systems, and in each of those cases, I had strange errors reported in the /var/log/messages file. That's not to say that a hard drive problem couldn't cause a lockup, but that hasn't been my past experience. With the lockups I'm currently seeing, I'm not seeing any error messages in the /var/log/messages file. At this point, I think anything and everything is suspect until proven otherwise. Thanks for the ideas! I'm jotting them all down. Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpjjYHl0iXDssISsRAvQFAJ4w8Q4A5Ai25Se0pgkoKdYWo9AFcACdF3Q4 bC9BhLL4vESujGOMSsA4R2o= =+CpT -----END PGP SIGNATURE-----

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Baris Hasdemir wrote: > On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: >>I'd rather not give up KDE, but I could try blackbox for a while to see >>what happens. > > It might also worth to disable hardware acceleration support. For > example you can use nv instead of nvidia, for a while. > > baris HI Baris, Thanks for the tip, but my video card is an older one based on the ATI Radeon 7500. Not everything in this machine is new. Thanks, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpjdrHl0iXDssISsRAoz4AJoCqnNFec1qMaYhLluDuyajGurPGQCfYbSi H4FuxFHjw6+5FVAfcaoWLjY= =3+wG -----END PGP SIGNATURE-----

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Walt Sawyer

Andy, Are you sure it's the hard drive? Try running Knoppix live CD for a while. If it's still running after days(max uptime)+5, then perhaps it's something else. If this message is redundant, ignore it! Walt On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Jeff Moyer wrote: > > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: > > > > andystewart> HI gang, > > > > andystewart> My dual Opteron machine is not happy. I cannot get more than > > andystewart> 7 straight days of uptime without getting a hard lock, > > andystewart> requiring a reboot. (My definition of hard lock is: machine > > andystewart> responds neither to keyboard input, mouse input, nor network > > andystewart> pings). > > > > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had > > andystewart> 3 tonight, and 3-4 on a previous occasion while running > > andystewart> OpenOffice). It makes no sense to me that an application run > > andystewart> as a normal user could lockup a machine. > > > > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an > > andystewart> "oops" when it hard locks - no dice. Do you know any other > > andystewart> tricks I could try to see if it is the kernel which is locking > > andystewart> up? I'm running SuSE's version of 2.6.8. > > > > Did you verify that NMIs are being delivered? After boot, cat > > /proc/interrupts and make sure the NMI line is non-zero. Also note that, > > at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 > > for Opterons (i.e. you shouldn't need to manually set it). > > HI Jeff, > > Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me > think they are NOT being delivered. How would I go about solving *that* > little problem? I did a "cat /proc/cmdline" to insure that I had > "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've > been seeking. > > > If the NMI watchdog works, it will print a message to the console. > > However, you will not see this if you are in X windows. Do you have a > > serial console hooked up, by any chance? I strongly suggest it if you have > > the means. > > I think I have a cable to which I could connect the serial port of the > Opteron to the serial port of another Linux box (and then use minicom or > some such terminal program). > > > > > Aside from this, if it is indeed a hard lockup, there is really nothing you > > can do (without purchasing other hardware to help debug the problem). > > Yeah, I was afraid of that. > > > > > Please give these suggestions a shot and let us know how it goes. > > Shall do - thanks, everybody! > > Oh, be advised that when you smash your fist on the keyboard after your > system locks up for the umpteenth time, that a lot of dead skin cells > will come flying upward out of the bowels of the keyboard. I would > recommend safety glasses. > > Later, > > Andy > > - -- > Andy Stewart, Founder > Worcester Linux Users' Group > Worcester, MA, USA > http://www.wlug.org > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.5 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > > iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV > u6O8n+PSW4M0I1MHXsH06Xo= > =ViVg > -----END PGP SIGNATURE----- > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > --

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Baris Hasdemir

On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: > I'd rather not give up KDE, but I could try blackbox for a while to see > what happens. It might also worth to disable hardware acceleration support. For example you can use nv instead of nvidia, for a while. baris

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote: > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: > > andystewart> HI gang, > > andystewart> My dual Opteron machine is not happy. I cannot get more than > andystewart> 7 straight days of uptime without getting a hard lock, > andystewart> requiring a reboot. (My definition of hard lock is: machine > andystewart> responds neither to keyboard input, mouse input, nor network > andystewart> pings). > > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had > andystewart> 3 tonight, and 3-4 on a previous occasion while running > andystewart> OpenOffice). It makes no sense to me that an application run > andystewart> as a normal user could lockup a machine. > > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an > andystewart> "oops" when it hard locks - no dice. Do you know any other > andystewart> tricks I could try to see if it is the kernel which is locking > andystewart> up? I'm running SuSE's version of 2.6.8. > > Did you verify that NMIs are being delivered? After boot, cat > /proc/interrupts and make sure the NMI line is non-zero. Also note that, > at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 > for Opterons (i.e. you shouldn't need to manually set it). HI Jeff, Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me think they are NOT being delivered. How would I go about solving *that* little problem? I did a "cat /proc/cmdline" to insure that I had "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've been seeking. > If the NMI watchdog works, it will print a message to the console. > However, you will not see this if you are in X windows. Do you have a > serial console hooked up, by any chance? I strongly suggest it if you have > the means. I think I have a cable to which I could connect the serial port of the Opteron to the serial port of another Linux box (and then use minicom or some such terminal program). > > Aside from this, if it is indeed a hard lockup, there is really nothing you > can do (without purchasing other hardware to help debug the problem). Yeah, I was afraid of that. > > Please give these suggestions a shot and let us know how it goes. Shall do - thanks, everybody! Oh, be advised that when you smash your fist on the keyboard after your system locks up for the umpteenth time, that a lot of dead skin cells will come flying upward out of the bowels of the keyboard. I would recommend safety glasses. Later, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV u6O8n+PSW4M0I1MHXsH06Xo= =ViVg -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Gary at home wrote: > Andy Stewart wrote: >>It makes no >>sense to me that an application run as a normal user could lockup a machine. > > Oh it's easy to crash a Linux machine with a fork-fest - any who can > fork a process can do it, so a rouge app can certainly do the same. > > What desktop are you running? I previously had similar problems with > Gnome on my Athlon XP under Gentoo that mysteriously stopped after a few > package updates...and I found no bug alerts for my symptoms at the time. > > Any chance you can run top or some other real-time system monitor when > the crash happens? > > -- Gary HI Gary, I am running KDE version 3.3.0 (SuSE 9.2). I could leave top running on a console to see if I can capture some useful info when the system locks up. I'll try that - thanks for the suggestion. I'd rather not give up KDE, but I could try blackbox for a while to see what happens. Thanks for the suggestions, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpipsHl0iXDssISsRAuPGAJ0T6JzfgqbGuWrB05uuJaPMohGytACeIbwJ mHYyFd2lDE8JbLQ4FAXC7Dk= =sdOC -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote: > ==> Regarding Re: [Wlug] HOWTO debug hard lockups; "Chuck R. Anderson" <cra(a)WPI.EDU> adds: > > cra> On Sun, Jun 05, 2005 at 10:35:13PM -0400, Andy Stewart wrote: >>>I've tried setting "nmi_watchdog=1" to see if I could get an "oops" when >>>it hard locks - no dice. Do you know any other tricks I could try to >>>see if it is the kernel which is locking up? I'm running SuSE's version >>>of 2.6.8. > > cra> Have you enabled the kernel Magic SysRq key and tried typing that when > cra> it locks up? > > cra> sysctl -w kernel.sysrq = 1 > > cra> When it locks up: > > cra> Alt-SysRq-? for help, then try dumping the PC, Tasks, etc. You may > cra> even be able to do an Emergency RO Mount and Sync before doing a Boot. > > This requires that you are not in X windows at the time of the lockup. See > my other email in this thead. HI all, The damn thing locked up on me *again* this evening, while trying to reply to this thread. On every occasion, I have been in X windows at the time of the lockup. I am either actively working or the screen saver is running, and the system locks up sometime in the middle of the night. What I can do is leave X windows running, but switch to a console before going to bed. If it hangs in the night, will I then be able to use the Alt-SysReq key sequences to get debug info? Thanks! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpinZHl0iXDssISsRAvwdAJ96My0XChSA5+kGI8XxzULjy27RMQCfVvey aSEYIUzjrxzfhr38eAs6q7Q= =+3IF -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 J Schonberg wrote: > > When the machine hung, did the num lock key still turn on and off the > led on the keyboard? If not, then other key sequences are probably > not working either. Is this a USB keyboard? > Hi Jason, I don't know if pressing the num lock key would cause the keyboard LED to toggle while the system is locked up. I'll try that next time the system locks. The keyboard is an old AT keyboard connected to my KVM switch, to which 3 other systems are connected via their keyboard/mouse connectors (not USB in any case). Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpidEHl0iXDssISsRAr5LAJwLAKrL84fdZw39XaYAcYwM8dTvUQCfQLDW lVQltv7fJj7tWFmpXC6l1es= =364+ -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Gary＠home

Andy Stewart wrote: > It makes no > sense to me that an application run as a normal user could lockup a machine. Oh it's easy to crash a Linux machine with a fork-fest - any who can fork a process can do it, so a rouge app can certainly do the same. What desktop are you running? I previously had similar problems with Gnome on my Athlon XP under Gentoo that mysteriously stopped after a few package updates...and I found no bug alerts for my symptoms at the time. Any chance you can run top or some other real-time system monitor when the crash happens? -- Gary

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by J Schonberg

--- Andy Stewart <andystewart(a)comcast.net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > Hi Chuck et al, > > The machine just locked up again (and it is much cooler today than > yesterday). I wasn't running OpenOffice this time. I tried > alt-printscreen-? and nothing happened. I tried > alt-printscreen-anything and nothing happened. crl-alt-delete, > ctrl-alt-backspace - no keyboard input at all was recognized as far > as I > can tell. > When the machine hung, did the num lock key still turn on and off the led on the keyboard? If not, then other key sequences are probably not working either. Is this a USB keyboard? Jason. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Chuck et al, The machine just locked up again (and it is much cooler today than yesterday). I wasn't running OpenOffice this time. I tried alt-printscreen-? and nothing happened. I tried alt-printscreen-anything and nothing happened. crl-alt-delete, ctrl-alt-backspace - no keyboard input at all was recognized as far as I can tell. Also, I found a lot of entries about this MB on the MSI Forum and it seems that the temp sensors register about 15% higher that reality. gkrellm allows me to tweak the readings, so I did. An independent temperature measurement confirms that the onboard sensors are registering about 15% high, and that I'm not really frying my CPUs as I had thought. At idle, my CPUs are around 43 degrees C. This makes this issue all the more puzzling. Other thoughts? I'm about to fire up an all night run of memtest. Thanks, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpQp+Hl0iXDssISsRAnBYAJ9WZDDB0G/y6lL31aYJDYPfNHJe1QCdHkSf k1E6p2ayY5rGt/+Lamb0VzU= =w7NU -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] Back issues of Linux Journal magazine available

by Mike Frysinger

On Monday 06 June 2005 08:10 pm, Andy Stewart wrote: > If nobody wants them by next Monday, I'll send them to recycling. who knows, maybe someone on ebay would want em ... throw em up for like a $1.00 :) -mike

June 7, 2005

Back issues of Linux Journal magazine available

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 HI everybody, I am cleaning out a closet and have some back issues of Linux Journal magazine that I'd like to give to somebody who wants them. They are all in great shape, all have been read, and (for you collectors) most probably do not have the mail in subscription cards (I like to use those as bookmarks). Aug 1995 (issue 16) - Dec 1995 All issues from 1996 All issues from 1997, incl Buyer's Guide All issues from 1998, incl Buyer's Guide All issues from 1999, incl "Enterprise Solutions" supplement If nobody wants them by next Monday, I'll send them to recycling. I'd be happy to bring them to the next WLUG meeting if somebody specifically says they want them. Later, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpOXZHl0iXDssISsRApXfAJ9UODjZ7meFzPZ/YesyBbsKDPBlUgCeM8Pa Ovvq9jm+aPlUetgGUSG94pQ= =WsE0 -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chuck R. Anderson wrote: > On Sun, Jun 05, 2005 at 10:35:13PM -0400, Andy Stewart wrote: >>I've tried setting "nmi_watchdog=1" to see if I could get an "oops" when >>it hard locks - no dice. Do you know any other tricks I could try to >>see if it is the kernel which is locking up? I'm running SuSE's version >>of 2.6.8. > > Have you enabled the kernel Magic SysRq key and tried typing that when > it locks up? > > sysctl -w kernel.sysrq = 1 > > When it locks up: > > Alt-SysRq-? for help, then try dumping the PC, Tasks, etc. You may > even be able to do an Emergency RO Mount and Sync before doing a Boot. > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > HI Chuck, That's a good idea which I've implemented per your suggestion. Now, I just have to wait for the next hang (hey, I could update the WLUG website with OpenOffice!). I'll keep you posted. Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpMefHl0iXDssISsRAnEsAJ9bcbaboREGjDKodlg4QS/I3+UB1ACfZF6x yGYCi38pKMmFAUyrE89aEfA= =r3Kq -----END PGP SIGNATURE-----

June 6, 2005

Re: [Wlug] HOWTO debug hard lockups

by Jeff Moyer

==> Regarding Re: [Wlug] HOWTO debug hard lockups; "Chuck R. Anderson" <cra(a)WPI.EDU> adds: cra> On Sun, Jun 05, 2005 at 10:35:13PM -0400, Andy Stewart wrote: >> I've tried setting "nmi_watchdog=1" to see if I could get an "oops" when >> it hard locks - no dice. Do you know any other tricks I could try to >> see if it is the kernel which is locking up? I'm running SuSE's version >> of 2.6.8. cra> Have you enabled the kernel Magic SysRq key and tried typing that when cra> it locks up? cra> sysctl -w kernel.sysrq = 1 cra> When it locks up: cra> Alt-SysRq-? for help, then try dumping the PC, Tasks, etc. You may cra> even be able to do an Emergency RO Mount and Sync before doing a Boot. This requires that you are not in X windows at the time of the lockup. See my other email in this thead. Also note that, due to no specific instances or facts to back up my statement, I don't like doing the 'Sync'. If the system is locked up, who is to say that you are going to write out valid data to the disk? I'd rather just reboot and let the journal recover things. -Jeff

June 6, 2005