- WLUG - wlug.mailman3.com

Re: [Wlug] HOWTO debug hard lockups

by Walt Sawyer

Andy, Are you sure it's the hard drive? Try running Knoppix live CD for a while. If it's still running after days(max uptime)+5, then perhaps it's something else. If this message is redundant, ignore it! Walt On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Jeff Moyer wrote: > > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: > > > > andystewart> HI gang, > > > > andystewart> My dual Opteron machine is not happy. I cannot get more than > > andystewart> 7 straight days of uptime without getting a hard lock, > > andystewart> requiring a reboot. (My definition of hard lock is: machine > > andystewart> responds neither to keyboard input, mouse input, nor network > > andystewart> pings). > > > > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had > > andystewart> 3 tonight, and 3-4 on a previous occasion while running > > andystewart> OpenOffice). It makes no sense to me that an application run > > andystewart> as a normal user could lockup a machine. > > > > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an > > andystewart> "oops" when it hard locks - no dice. Do you know any other > > andystewart> tricks I could try to see if it is the kernel which is locking > > andystewart> up? I'm running SuSE's version of 2.6.8. > > > > Did you verify that NMIs are being delivered? After boot, cat > > /proc/interrupts and make sure the NMI line is non-zero. Also note that, > > at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 > > for Opterons (i.e. you shouldn't need to manually set it). > > HI Jeff, > > Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me > think they are NOT being delivered. How would I go about solving *that* > little problem? I did a "cat /proc/cmdline" to insure that I had > "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've > been seeking. > > > If the NMI watchdog works, it will print a message to the console. > > However, you will not see this if you are in X windows. Do you have a > > serial console hooked up, by any chance? I strongly suggest it if you have > > the means. > > I think I have a cable to which I could connect the serial port of the > Opteron to the serial port of another Linux box (and then use minicom or > some such terminal program). > > > > > Aside from this, if it is indeed a hard lockup, there is really nothing you > > can do (without purchasing other hardware to help debug the problem). > > Yeah, I was afraid of that. > > > > > Please give these suggestions a shot and let us know how it goes. > > Shall do - thanks, everybody! > > Oh, be advised that when you smash your fist on the keyboard after your > system locks up for the umpteenth time, that a lot of dead skin cells > will come flying upward out of the bowels of the keyboard. I would > recommend safety glasses. > > Later, > > Andy > > - -- > Andy Stewart, Founder > Worcester Linux Users' Group > Worcester, MA, USA > http://www.wlug.org > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.5 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > > iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV > u6O8n+PSW4M0I1MHXsH06Xo= > =ViVg > -----END PGP SIGNATURE----- > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > --

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Baris Hasdemir

On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: > I'd rather not give up KDE, but I could try blackbox for a while to see > what happens. It might also worth to disable hardware acceleration support. For example you can use nv instead of nvidia, for a while. baris

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote: > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: > > andystewart> HI gang, > > andystewart> My dual Opteron machine is not happy. I cannot get more than > andystewart> 7 straight days of uptime without getting a hard lock, > andystewart> requiring a reboot. (My definition of hard lock is: machine > andystewart> responds neither to keyboard input, mouse input, nor network > andystewart> pings). > > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had > andystewart> 3 tonight, and 3-4 on a previous occasion while running > andystewart> OpenOffice). It makes no sense to me that an application run > andystewart> as a normal user could lockup a machine. > > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an > andystewart> "oops" when it hard locks - no dice. Do you know any other > andystewart> tricks I could try to see if it is the kernel which is locking > andystewart> up? I'm running SuSE's version of 2.6.8. > > Did you verify that NMIs are being delivered? After boot, cat > /proc/interrupts and make sure the NMI line is non-zero. Also note that, > at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 > for Opterons (i.e. you shouldn't need to manually set it). HI Jeff, Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me think they are NOT being delivered. How would I go about solving *that* little problem? I did a "cat /proc/cmdline" to insure that I had "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've been seeking. > If the NMI watchdog works, it will print a message to the console. > However, you will not see this if you are in X windows. Do you have a > serial console hooked up, by any chance? I strongly suggest it if you have > the means. I think I have a cable to which I could connect the serial port of the Opteron to the serial port of another Linux box (and then use minicom or some such terminal program). > > Aside from this, if it is indeed a hard lockup, there is really nothing you > can do (without purchasing other hardware to help debug the problem). Yeah, I was afraid of that. > > Please give these suggestions a shot and let us know how it goes. Shall do - thanks, everybody! Oh, be advised that when you smash your fist on the keyboard after your system locks up for the umpteenth time, that a lot of dead skin cells will come flying upward out of the bowels of the keyboard. I would recommend safety glasses. Later, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV u6O8n+PSW4M0I1MHXsH06Xo= =ViVg -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Gary at home wrote: > Andy Stewart wrote: >>It makes no >>sense to me that an application run as a normal user could lockup a machine. > > Oh it's easy to crash a Linux machine with a fork-fest - any who can > fork a process can do it, so a rouge app can certainly do the same. > > What desktop are you running? I previously had similar problems with > Gnome on my Athlon XP under Gentoo that mysteriously stopped after a few > package updates...and I found no bug alerts for my symptoms at the time. > > Any chance you can run top or some other real-time system monitor when > the crash happens? > > -- Gary HI Gary, I am running KDE version 3.3.0 (SuSE 9.2). I could leave top running on a console to see if I can capture some useful info when the system locks up. I'll try that - thanks for the suggestion. I'd rather not give up KDE, but I could try blackbox for a while to see what happens. Thanks for the suggestions, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpipsHl0iXDssISsRAuPGAJ0T6JzfgqbGuWrB05uuJaPMohGytACeIbwJ mHYyFd2lDE8JbLQ4FAXC7Dk= =sdOC -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote: > ==> Regarding Re: [Wlug] HOWTO debug hard lockups; "Chuck R. Anderson" <cra(a)WPI.EDU> adds: > > cra> On Sun, Jun 05, 2005 at 10:35:13PM -0400, Andy Stewart wrote: >>>I've tried setting "nmi_watchdog=1" to see if I could get an "oops" when >>>it hard locks - no dice. Do you know any other tricks I could try to >>>see if it is the kernel which is locking up? I'm running SuSE's version >>>of 2.6.8. > > cra> Have you enabled the kernel Magic SysRq key and tried typing that when > cra> it locks up? > > cra> sysctl -w kernel.sysrq = 1 > > cra> When it locks up: > > cra> Alt-SysRq-? for help, then try dumping the PC, Tasks, etc. You may > cra> even be able to do an Emergency RO Mount and Sync before doing a Boot. > > This requires that you are not in X windows at the time of the lockup. See > my other email in this thead. HI all, The damn thing locked up on me *again* this evening, while trying to reply to this thread. On every occasion, I have been in X windows at the time of the lockup. I am either actively working or the screen saver is running, and the system locks up sometime in the middle of the night. What I can do is leave X windows running, but switch to a console before going to bed. If it hangs in the night, will I then be able to use the Alt-SysReq key sequences to get debug info? Thanks! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpinZHl0iXDssISsRAvwdAJ96My0XChSA5+kGI8XxzULjy27RMQCfVvey aSEYIUzjrxzfhr38eAs6q7Q= =+3IF -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 J Schonberg wrote: > > When the machine hung, did the num lock key still turn on and off the > led on the keyboard? If not, then other key sequences are probably > not working either. Is this a USB keyboard? > Hi Jason, I don't know if pressing the num lock key would cause the keyboard LED to toggle while the system is locked up. I'll try that next time the system locks. The keyboard is an old AT keyboard connected to my KVM switch, to which 3 other systems are connected via their keyboard/mouse connectors (not USB in any case). Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpidEHl0iXDssISsRAr5LAJwLAKrL84fdZw39XaYAcYwM8dTvUQCfQLDW lVQltv7fJj7tWFmpXC6l1es= =364+ -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Gary＠home

Andy Stewart wrote: > It makes no > sense to me that an application run as a normal user could lockup a machine. Oh it's easy to crash a Linux machine with a fork-fest - any who can fork a process can do it, so a rouge app can certainly do the same. What desktop are you running? I previously had similar problems with Gnome on my Athlon XP under Gentoo that mysteriously stopped after a few package updates...and I found no bug alerts for my symptoms at the time. Any chance you can run top or some other real-time system monitor when the crash happens? -- Gary

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by J Schonberg

--- Andy Stewart <andystewart(a)comcast.net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > Hi Chuck et al, > > The machine just locked up again (and it is much cooler today than > yesterday). I wasn't running OpenOffice this time. I tried > alt-printscreen-? and nothing happened. I tried > alt-printscreen-anything and nothing happened. crl-alt-delete, > ctrl-alt-backspace - no keyboard input at all was recognized as far > as I > can tell. > When the machine hung, did the num lock key still turn on and off the led on the keyboard? If not, then other key sequences are probably not working either. Is this a USB keyboard? Jason. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Chuck et al, The machine just locked up again (and it is much cooler today than yesterday). I wasn't running OpenOffice this time. I tried alt-printscreen-? and nothing happened. I tried alt-printscreen-anything and nothing happened. crl-alt-delete, ctrl-alt-backspace - no keyboard input at all was recognized as far as I can tell. Also, I found a lot of entries about this MB on the MSI Forum and it seems that the temp sensors register about 15% higher that reality. gkrellm allows me to tweak the readings, so I did. An independent temperature measurement confirms that the onboard sensors are registering about 15% high, and that I'm not really frying my CPUs as I had thought. At idle, my CPUs are around 43 degrees C. This makes this issue all the more puzzling. Other thoughts? I'm about to fire up an all night run of memtest. Thanks, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpQp+Hl0iXDssISsRAnBYAJ9WZDDB0G/y6lL31aYJDYPfNHJe1QCdHkSf k1E6p2ayY5rGt/+Lamb0VzU= =w7NU -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] Back issues of Linux Journal magazine available

by Mike Frysinger

On Monday 06 June 2005 08:10 pm, Andy Stewart wrote: > If nobody wants them by next Monday, I'll send them to recycling. who knows, maybe someone on ebay would want em ... throw em up for like a $1.00 :) -mike

June 7, 2005