- WLUG - wlug.mailman3.com

Re: [Wlug] HOWTO debug hard lockups

by John Stoffel

Andy, The best thing to do here is to run a serial console to another one of your systems and use THAT to try and catch the issue. You can then do a SysRq from the remote serial console and/or catch any error messages. Also, are there any messages in dmesg or /var/log/messages? How does the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see if the problem repeats. John

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Jeff Moyer

==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 andystewart> Jeff Moyer wrote: >> ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart >> <andystewart(a)comcast.net> adds: >> andystewart> HI gang, >> andystewart> My dual Opteron machine is not happy. I cannot get more than andystewart> 7 straight days of uptime without getting a hard lock, andystewart> requiring a reboot. (My definition of hard lock is: machine andystewart> responds neither to keyboard input, mouse input, nor network andystewart> pings). >> andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had andystewart> 3 tonight, and 3-4 on a previous occasion while running andystewart> OpenOffice). It makes no sense to me that an application run andystewart> as a normal user could lockup a machine. >> andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an andystewart> "oops" when it hard locks - no dice. Do you know any other andystewart> tricks I could try to see if it is the kernel which is locking andystewart> up? I'm running SuSE's version of 2.6.8. >> Did you verify that NMIs are being delivered? After boot, cat >> /proc/interrupts and make sure the NMI line is non-zero. Also note >> that, at least with upstream and Red Hat kernels, the nmi_watchdog >> defaults to 1 for Opterons (i.e. you shouldn't need to manually set it). andystewart> HI Jeff, andystewart> Well, this is weird, I'm seeing a ZERO count for NMIs, so that andystewart> makes me think they are NOT being delivered. How would I go andystewart> about solving *that* little problem? I did a "cat Well, you can try booting with nmi_watchdog=2. This will try to use the local APIC to deliver nmi's, but I haven't actually seen a dual processor system that required this (all of them I've seen work with nmi_watchdog=1). It is worth a try, however. andystewart> /proc/cmdline" to insure that I had "nmi_watchdog=1" and andystewart> indeed it is there. Perhaps this is the clue we've been andystewart> seeking. Well, it only tells you that the nmi_watchdog won't trigger. We still have no insight into what the problem might actually be. >> If the NMI watchdog works, it will print a message to the console. >> However, you will not see this if you are in X windows. Do you have a >> serial console hooked up, by any chance? I strongly suggest it if you >> have the means. andystewart> I think I have a cable to which I could connect the serial andystewart> port of the Opteron to the serial port of another Linux box andystewart> (and then use minicom or some such terminal program). >> Aside from this, if it is indeed a hard lockup, there is really nothing >> you can do (without purchasing other hardware to help debug the >> problem). andystewart> Yeah, I was afraid of that. >> Please give these suggestions a shot and let us know how it goes. andystewart> Shall do - thanks, everybody! andystewart> Oh, be advised that when you smash your fist on the keyboard andystewart> after your system locks up for the umpteenth time, that a lot andystewart> of dead skin cells will come flying upward out of the bowels andystewart> of the keyboard. I would recommend safety glasses. Noted. =) -Jeff

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Jeff Moyer

==> Regarding Re: [Wlug] HOWTO debug hard lockups; Walt Sawyer <wsawyer(a)norfolk-county.com> adds: wsawyer> Andy, Are you sure it's the hard drive? Try running Knoppix live wsawyer> CD for a while. If it's still running after days(max uptime)+5, wsawyer> then perhaps it's something else. This changes things entirely. He'll be running a different kernel, so this test is not useful, in my opinion. -Jeff wsawyer> If this message is redundant, ignore it! Walt wsawyer> On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wrote: >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >> >> Jeff Moyer wrote: > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy >> Stewart <andystewart(a)comcast.net> adds: >> > >> > andystewart> HI gang, >> > >> > andystewart> My dual Opteron machine is not happy. I cannot get more >> than > andystewart> 7 straight days of uptime without getting a hard >> lock, > andystewart> requiring a reboot. (My definition of hard lock >> is: machine > andystewart> responds neither to keyboard input, mouse >> input, nor network > andystewart> pings). >> > >> > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I >> had > andystewart> 3 tonight, and 3-4 on a previous occasion while >> running > andystewart> OpenOffice). It makes no sense to me that an >> application run > andystewart> as a normal user could lockup a machine. >> > >> > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get >> an > andystewart> "oops" when it hard locks - no dice. Do you know any >> other > andystewart> tricks I could try to see if it is the kernel which >> is locking > andystewart> up? I'm running SuSE's version of 2.6.8. >> > >> > Did you verify that NMIs are being delivered? After boot, cat > >> /proc/interrupts and make sure the NMI line is non-zero. Also note >> that, > at least with upstream and Red Hat kernels, the nmi_watchdog >> defaults to 1 > for Opterons (i.e. you shouldn't need to manually set >> it). >> >> HI Jeff, >> >> Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me >> think they are NOT being delivered. How would I go about solving *that* >> little problem? I did a "cat /proc/cmdline" to insure that I had >> "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've >> been seeking. >> >> > If the NMI watchdog works, it will print a message to the console. > >> However, you will not see this if you are in X windows. Do you have a > >> serial console hooked up, by any chance? I strongly suggest it if you >> have > the means. >> >> I think I have a cable to which I could connect the serial port of the >> Opteron to the serial port of another Linux box (and then use minicom or >> some such terminal program). >> >> > >> > Aside from this, if it is indeed a hard lockup, there is really >> nothing you > can do (without purchasing other hardware to help debug >> the problem). >> >> Yeah, I was afraid of that. >> >> > >> > Please give these suggestions a shot and let us know how it goes. >> >> Shall do - thanks, everybody! >> >> Oh, be advised that when you smash your fist on the keyboard after your >> system locks up for the umpteenth time, that a lot of dead skin cells >> will come flying upward out of the bowels of the keyboard. I would >> recommend safety glasses. >> >> Later, >> >> Andy >> >> - -- >> Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA >> http://www.wlug.org >> >> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: >> Using GnuPG with Thunderbird - http://enigmail.mozdev.org >> >> iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV >> u6O8n+PSW4M0I1MHXsH06Xo= =ViVg -----END PGP SIGNATURE----- >> _______________________________________________ Wlug mailing list >> Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug >> wsawyer> -- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN"> wsawyer> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; wsawyer> CHARSET=UTF-8"> <META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2"> wsawyer> </HEAD> <BODY> Andy, Are you sure it's the hard drive?  wsawyer> Try running Knoppix live CD for a while.  If it's still wsawyer> running after days(max uptime)+5, then perhaps it's something wsawyer> else. If this message is redundant, ignore it! wsawyer> Walt On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wsawyer> wrote: <BLOCKQUOTE TYPE=CITE> <PRE> COLOR="#000000">-----BEGIN PGP SIGNED MESSAGE----- COLOR="#000000">Hash: SHA1 wsawyer> Jeff Moyer wrote: COLOR="#000000">> ==> Regarding [Wlug] HOWTO debug hard wsawyer> lockups; Andy Stewart <<A wsawyer> HREF="mailto:andystewart@comcast.net">andystewart(a)comcast.net</A>> wsawyer> adds: > COLOR="#000000">> andystewart> HI gang, COLOR="#000000">> > wsawyer> andystewart> My dual Opteron machine is not happy. I cannot wsawyer> get more than > andystewart> 7 wsawyer> straight days of uptime without getting a hard lock, COLOR="#000000">> andystewart> requiring a reboot. (My wsawyer> definition of hard lock is: machine COLOR="#000000">> andystewart> responds neither to keyboard wsawyer> input, mouse input, nor network > wsawyer> andystewart> pings). > wsawyer> > andystewart> I can stimulate hard wsawyer> locks by running OpenOffice 1.1.3 (I had COLOR="#000000">> andystewart> 3 tonight, and 3-4 on a wsawyer> previous occasion while running > wsawyer> andystewart> OpenOffice). It makes no sense to me that an wsawyer> application run > andystewart> wsawyer> as a normal user could lockup a machine. COLOR="#000000">> > wsawyer> andystewart> I've tried setting "nmi_watchdog=1" to wsawyer> see if I could get an > wsawyer> andystewart> "oops" when it hard locks - no dice. Do wsawyer> you know any other > wsawyer> andystewart> tricks I could try to see if it is the kernel wsawyer> which is locking > andystewart> wsawyer> up? I'm running SuSE's version of 2.6.8. COLOR="#000000">> > Did you wsawyer> verify that NMIs are being delivered? After boot, cat wsawyer> > /proc/interrupts and make sure the NMI wsawyer> line is non-zero. Also note that, COLOR="#000000">> at least with upstream and Red Hat kernels, wsawyer> the nmi_watchdog defaults to 1 > wsawyer> for Opterons (i.e. you shouldn't need to manually set it). wsawyer> HI Jeff, wsawyer> Well, this is weird, I'm seeing a ZERO count wsawyer> for NMIs, so that makes me think they wsawyer> are NOT being delivered. How would I go about solving wsawyer> *that* little problem? I did a wsawyer> "cat /proc/cmdline" to insure that I had COLOR="#000000">"nmi_watchdog=1" and indeed it is there. wsawyer> Perhaps this is the clue we've been wsawyer> seeking. wsawyer> > If the NMI watchdog works, it will wsawyer> print a message to the console. > wsawyer> However, you will not see this if you are in X windows. Do you wsawyer> have a > serial console hooked up, wsawyer> by any chance? I strongly suggest it if you have COLOR="#000000">> the means. wsawyer> I think I have a cable to which I could wsawyer> connect the serial port of the COLOR="#000000">Opteron to the serial port of another Linux box wsawyer> (and then use minicom or some such wsawyer> terminal program). wsawyer> > > wsawyer> Aside from this, if it is indeed a hard lockup, there is really wsawyer> nothing you > can do (without wsawyer> purchasing other hardware to help debug the problem). wsawyer> Yeah, I was afraid of that. wsawyer> > > wsawyer> Please give these suggestions a shot and let us know how it wsawyer> goes. wsawyer> Shall do - thanks, everybody! wsawyer> Oh, be advised that when you smash your fist wsawyer> on the keyboard after your system wsawyer> locks up for the umpteenth time, that a lot of dead skin wsawyer> cells will come flying upward out of wsawyer> the bowels of the keyboard. I would COLOR="#000000">recommend safety glasses. wsawyer> Later, wsawyer> Andy wsawyer> - -- Andy wsawyer> Stewart, Founder Worcester Linux wsawyer> Users' Group Worcester, MA, wsawyer> USA <A wsawyer> HREF="http://www.wlug.org">http://www.wlug.org</A> wsawyer> -----BEGIN PGP SIGNATURE----- COLOR="#000000">Version: GnuPG v1.2.5 (GNU/Linux) COLOR="#000000">Comment: Using GnuPG with Thunderbird - <A wsawyer> HREF="http://enigmail.mozdev.org">http://enigmail.mozdev.org</A> wsawyer> COLOR="#000000">iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV wsawyer> u6O8n+PSW4M0I1MHXsH06Xo= COLOR="#000000">=ViVg -----END PGP wsawyer> SIGNATURE----- COLOR="#000000">_______________________________________________ wsawyer> Wlug mailing list COLOR="#000000"><A wsawyer> HREF="mailto:Wlug@mail.wlug.org">Wlug(a)mail.wlug.org</A> wsawyer> <A wsawyer> HREF="http://mail.wlug.org/mailman/listinfo/wlug">http://mail.wlug.org/mailman/listinfo/wlug</A> wsawyer> </PRE> </BLOCKQUOTE> <TABLE CELLSPACING="0" CELLPADDING="0" wsawyer> WIDTH="100%"> <TR> <TD> <PRE> -- </PRE> </TD> </TR> </TABLE> wsawyer> </BODY> </HTML> _______________________________________________ wsawyer> Wlug mailing list Wlug(a)mail.wlug.org wsawyer> http://mail.wlug.org/mailman/listinfo/wlug

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Gary Hanley

On Tue, 7 Jun 2005, Andy Stewart wrote: > I'd rather not give up KDE, but I could try blackbox for a while to see > what happens. Well, my guess is that it is a hardware issue How about the MB settings, like the memory timings/clock ratios? Sometimes this can cause problems if they are set too aggressively. Usually there are one-off options in the BIOS menu for stable settings and speed settings so you don't have to tweak everything. Anything there you can check? Maybe down-shift your settings a little and see if the problem goes away. I haven't had the pleasure of working with a dual system every so I'm not familiar with any related BIOS settings you may need to modify. -- Gary

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Baris Hasdemir wrote: > On Tue, 2005-06-07 at 20:10 -0400, Andy Stewart wrote: >>-----BEGIN PGP SIGNED MESSAGE----- >>Hash: SHA1 >> >>Baris Hasdemir wrote: >>>On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: >>>>I'd rather not give up KDE, but I could try blackbox for a while to see >>>>what happens. >>>It might also worth to disable hardware acceleration support. For >>>example you can use nv instead of nvidia, for a while. >>> >>>baris >>HI Baris, >> >>Thanks for the tip, but my video card is an older one based on the ATI >>Radeon 7500. Not everything in this machine is new. >> >>Thanks, > > Are you using ati or radeon module? if using latter you have the > hardware acceleration support, if I recall correctly. > HI Baris, Hmmm, let me check. It appears that I'm using the radeon module. I'll make a note to try disabling hardware acceleration support for the video card. Thanks for the idea! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpl32Hl0iXDssISsRAuKKAJ9nkqIT8By9WV4h9htjaiy7Sb/KpACbBmwF jInvBZ/LKPzIlezoXl7Bc+k= =qs7r -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Baris Hasdemir

On Tue, 2005-06-07 at 20:10 -0400, Andy Stewart wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Baris Hasdemir wrote: > > On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: > >>I'd rather not give up KDE, but I could try blackbox for a while to see > >>what happens. > > > > It might also worth to disable hardware acceleration support. For > > example you can use nv instead of nvidia, for a while. > > > > baris > > HI Baris, > > Thanks for the tip, but my video card is an older one based on the ATI > Radeon 7500. Not everything in this machine is new. > > Thanks, Are you using ati or radeon module? if using latter you have the hardware acceleration support, if I recall correctly. baris

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Walt Sawyer wrote: > Andy, > Are you sure it's the hard drive? Try running Knoppix live CD for a > while. If it's still running after days(max uptime)+5, then perhaps > it's something else. > > If this message is redundant, ignore it! > Walt HI Walt, I have suffered from hard drive problems on previous systems, and in each of those cases, I had strange errors reported in the /var/log/messages file. That's not to say that a hard drive problem couldn't cause a lockup, but that hasn't been my past experience. With the lockups I'm currently seeing, I'm not seeing any error messages in the /var/log/messages file. At this point, I think anything and everything is suspect until proven otherwise. Thanks for the ideas! I'm jotting them all down. Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpjjYHl0iXDssISsRAvQFAJ4w8Q4A5Ai25Se0pgkoKdYWo9AFcACdF3Q4 bC9BhLL4vESujGOMSsA4R2o= =+CpT -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Baris Hasdemir wrote: > On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: >>I'd rather not give up KDE, but I could try blackbox for a while to see >>what happens. > > It might also worth to disable hardware acceleration support. For > example you can use nv instead of nvidia, for a while. > > baris HI Baris, Thanks for the tip, but my video card is an older one based on the ATI Radeon 7500. Not everything in this machine is new. Thanks, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpjdrHl0iXDssISsRAoz4AJoCqnNFec1qMaYhLluDuyajGurPGQCfYbSi H4FuxFHjw6+5FVAfcaoWLjY= =3+wG -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Walt Sawyer

Andy, Are you sure it's the hard drive? Try running Knoppix live CD for a while. If it's still running after days(max uptime)+5, then perhaps it's something else. If this message is redundant, ignore it! Walt On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Jeff Moyer wrote: > > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: > > > > andystewart> HI gang, > > > > andystewart> My dual Opteron machine is not happy. I cannot get more than > > andystewart> 7 straight days of uptime without getting a hard lock, > > andystewart> requiring a reboot. (My definition of hard lock is: machine > > andystewart> responds neither to keyboard input, mouse input, nor network > > andystewart> pings). > > > > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had > > andystewart> 3 tonight, and 3-4 on a previous occasion while running > > andystewart> OpenOffice). It makes no sense to me that an application run > > andystewart> as a normal user could lockup a machine. > > > > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an > > andystewart> "oops" when it hard locks - no dice. Do you know any other > > andystewart> tricks I could try to see if it is the kernel which is locking > > andystewart> up? I'm running SuSE's version of 2.6.8. > > > > Did you verify that NMIs are being delivered? After boot, cat > > /proc/interrupts and make sure the NMI line is non-zero. Also note that, > > at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 > > for Opterons (i.e. you shouldn't need to manually set it). > > HI Jeff, > > Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me > think they are NOT being delivered. How would I go about solving *that* > little problem? I did a "cat /proc/cmdline" to insure that I had > "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've > been seeking. > > > If the NMI watchdog works, it will print a message to the console. > > However, you will not see this if you are in X windows. Do you have a > > serial console hooked up, by any chance? I strongly suggest it if you have > > the means. > > I think I have a cable to which I could connect the serial port of the > Opteron to the serial port of another Linux box (and then use minicom or > some such terminal program). > > > > > Aside from this, if it is indeed a hard lockup, there is really nothing you > > can do (without purchasing other hardware to help debug the problem). > > Yeah, I was afraid of that. > > > > > Please give these suggestions a shot and let us know how it goes. > > Shall do - thanks, everybody! > > Oh, be advised that when you smash your fist on the keyboard after your > system locks up for the umpteenth time, that a lot of dead skin cells > will come flying upward out of the bowels of the keyboard. I would > recommend safety glasses. > > Later, > > Andy > > - -- > Andy Stewart, Founder > Worcester Linux Users' Group > Worcester, MA, USA > http://www.wlug.org > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.5 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > > iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV > u6O8n+PSW4M0I1MHXsH06Xo= > =ViVg > -----END PGP SIGNATURE----- > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > --

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Baris Hasdemir

On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: > I'd rather not give up KDE, but I could try blackbox for a while to see > what happens. It might also worth to disable hardware acceleration support. For example you can use nv instead of nvidia, for a while. baris

June 7, 2005