June 2005 - WLUG - wlug.mailman3.com

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Baris Hasdemir wrote: > On Tue, 2005-06-07 at 20:10 -0400, Andy Stewart wrote: >>-----BEGIN PGP SIGNED MESSAGE----- >>Hash: SHA1 >> >>Baris Hasdemir wrote: >>>On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: >>>>I'd rather not give up KDE, but I could try blackbox for a while to see >>>>what happens. >>>It might also worth to disable hardware acceleration support. For >>>example you can use nv instead of nvidia, for a while. >>> >>>baris >>HI Baris, >> >>Thanks for the tip, but my video card is an older one based on the ATI >>Radeon 7500. Not everything in this machine is new. >> >>Thanks, > > Are you using ati or radeon module? if using latter you have the > hardware acceleration support, if I recall correctly. > HI Baris, Hmmm, let me check. It appears that I'm using the radeon module. I'll make a note to try disabling hardware acceleration support for the video card. Thanks for the idea! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpl32Hl0iXDssISsRAuKKAJ9nkqIT8By9WV4h9htjaiy7Sb/KpACbBmwF jInvBZ/LKPzIlezoXl7Bc+k= =qs7r -----END PGP SIGNATURE-----

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Baris Hasdemir

On Tue, 2005-06-07 at 20:10 -0400, Andy Stewart wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Baris Hasdemir wrote: > > On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: > >>I'd rather not give up KDE, but I could try blackbox for a while to see > >>what happens. > > > > It might also worth to disable hardware acceleration support. For > > example you can use nv instead of nvidia, for a while. > > > > baris > > HI Baris, > > Thanks for the tip, but my video card is an older one based on the ATI > Radeon 7500. Not everything in this machine is new. > > Thanks, Are you using ati or radeon module? if using latter you have the hardware acceleration support, if I recall correctly. baris

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Walt Sawyer wrote: > Andy, > Are you sure it's the hard drive? Try running Knoppix live CD for a > while. If it's still running after days(max uptime)+5, then perhaps > it's something else. > > If this message is redundant, ignore it! > Walt HI Walt, I have suffered from hard drive problems on previous systems, and in each of those cases, I had strange errors reported in the /var/log/messages file. That's not to say that a hard drive problem couldn't cause a lockup, but that hasn't been my past experience. With the lockups I'm currently seeing, I'm not seeing any error messages in the /var/log/messages file. At this point, I think anything and everything is suspect until proven otherwise. Thanks for the ideas! I'm jotting them all down. Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpjjYHl0iXDssISsRAvQFAJ4w8Q4A5Ai25Se0pgkoKdYWo9AFcACdF3Q4 bC9BhLL4vESujGOMSsA4R2o= =+CpT -----END PGP SIGNATURE-----

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Baris Hasdemir wrote: > On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: >>I'd rather not give up KDE, but I could try blackbox for a while to see >>what happens. > > It might also worth to disable hardware acceleration support. For > example you can use nv instead of nvidia, for a while. > > baris HI Baris, Thanks for the tip, but my video card is an older one based on the ATI Radeon 7500. Not everything in this machine is new. Thanks, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpjdrHl0iXDssISsRAoz4AJoCqnNFec1qMaYhLluDuyajGurPGQCfYbSi H4FuxFHjw6+5FVAfcaoWLjY= =3+wG -----END PGP SIGNATURE-----

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Walt Sawyer

Andy, Are you sure it's the hard drive? Try running Knoppix live CD for a while. If it's still running after days(max uptime)+5, then perhaps it's something else. If this message is redundant, ignore it! Walt On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Jeff Moyer wrote: > > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: > > > > andystewart> HI gang, > > > > andystewart> My dual Opteron machine is not happy. I cannot get more than > > andystewart> 7 straight days of uptime without getting a hard lock, > > andystewart> requiring a reboot. (My definition of hard lock is: machine > > andystewart> responds neither to keyboard input, mouse input, nor network > > andystewart> pings). > > > > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had > > andystewart> 3 tonight, and 3-4 on a previous occasion while running > > andystewart> OpenOffice). It makes no sense to me that an application run > > andystewart> as a normal user could lockup a machine. > > > > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an > > andystewart> "oops" when it hard locks - no dice. Do you know any other > > andystewart> tricks I could try to see if it is the kernel which is locking > > andystewart> up? I'm running SuSE's version of 2.6.8. > > > > Did you verify that NMIs are being delivered? After boot, cat > > /proc/interrupts and make sure the NMI line is non-zero. Also note that, > > at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 > > for Opterons (i.e. you shouldn't need to manually set it). > > HI Jeff, > > Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me > think they are NOT being delivered. How would I go about solving *that* > little problem? I did a "cat /proc/cmdline" to insure that I had > "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've > been seeking. > > > If the NMI watchdog works, it will print a message to the console. > > However, you will not see this if you are in X windows. Do you have a > > serial console hooked up, by any chance? I strongly suggest it if you have > > the means. > > I think I have a cable to which I could connect the serial port of the > Opteron to the serial port of another Linux box (and then use minicom or > some such terminal program). > > > > > Aside from this, if it is indeed a hard lockup, there is really nothing you > > can do (without purchasing other hardware to help debug the problem). > > Yeah, I was afraid of that. > > > > > Please give these suggestions a shot and let us know how it goes. > > Shall do - thanks, everybody! > > Oh, be advised that when you smash your fist on the keyboard after your > system locks up for the umpteenth time, that a lot of dead skin cells > will come flying upward out of the bowels of the keyboard. I would > recommend safety glasses. > > Later, > > Andy > > - -- > Andy Stewart, Founder > Worcester Linux Users' Group > Worcester, MA, USA > http://www.wlug.org > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.5 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > > iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV > u6O8n+PSW4M0I1MHXsH06Xo= > =ViVg > -----END PGP SIGNATURE----- > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > --

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Baris Hasdemir

On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: > I'd rather not give up KDE, but I could try blackbox for a while to see > what happens. It might also worth to disable hardware acceleration support. For example you can use nv instead of nvidia, for a while. baris

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote: > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: > > andystewart> HI gang, > > andystewart> My dual Opteron machine is not happy. I cannot get more than > andystewart> 7 straight days of uptime without getting a hard lock, > andystewart> requiring a reboot. (My definition of hard lock is: machine > andystewart> responds neither to keyboard input, mouse input, nor network > andystewart> pings). > > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had > andystewart> 3 tonight, and 3-4 on a previous occasion while running > andystewart> OpenOffice). It makes no sense to me that an application run > andystewart> as a normal user could lockup a machine. > > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an > andystewart> "oops" when it hard locks - no dice. Do you know any other > andystewart> tricks I could try to see if it is the kernel which is locking > andystewart> up? I'm running SuSE's version of 2.6.8. > > Did you verify that NMIs are being delivered? After boot, cat > /proc/interrupts and make sure the NMI line is non-zero. Also note that, > at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 > for Opterons (i.e. you shouldn't need to manually set it). HI Jeff, Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me think they are NOT being delivered. How would I go about solving *that* little problem? I did a "cat /proc/cmdline" to insure that I had "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've been seeking. > If the NMI watchdog works, it will print a message to the console. > However, you will not see this if you are in X windows. Do you have a > serial console hooked up, by any chance? I strongly suggest it if you have > the means. I think I have a cable to which I could connect the serial port of the Opteron to the serial port of another Linux box (and then use minicom or some such terminal program). > > Aside from this, if it is indeed a hard lockup, there is really nothing you > can do (without purchasing other hardware to help debug the problem). Yeah, I was afraid of that. > > Please give these suggestions a shot and let us know how it goes. Shall do - thanks, everybody! Oh, be advised that when you smash your fist on the keyboard after your system locks up for the umpteenth time, that a lot of dead skin cells will come flying upward out of the bowels of the keyboard. I would recommend safety glasses. Later, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV u6O8n+PSW4M0I1MHXsH06Xo= =ViVg -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Gary at home wrote: > Andy Stewart wrote: >>It makes no >>sense to me that an application run as a normal user could lockup a machine. > > Oh it's easy to crash a Linux machine with a fork-fest - any who can > fork a process can do it, so a rouge app can certainly do the same. > > What desktop are you running? I previously had similar problems with > Gnome on my Athlon XP under Gentoo that mysteriously stopped after a few > package updates...and I found no bug alerts for my symptoms at the time. > > Any chance you can run top or some other real-time system monitor when > the crash happens? > > -- Gary HI Gary, I am running KDE version 3.3.0 (SuSE 9.2). I could leave top running on a console to see if I can capture some useful info when the system locks up. I'll try that - thanks for the suggestion. I'd rather not give up KDE, but I could try blackbox for a while to see what happens. Thanks for the suggestions, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpipsHl0iXDssISsRAuPGAJ0T6JzfgqbGuWrB05uuJaPMohGytACeIbwJ mHYyFd2lDE8JbLQ4FAXC7Dk= =sdOC -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote: > ==> Regarding Re: [Wlug] HOWTO debug hard lockups; "Chuck R. Anderson" <cra(a)WPI.EDU> adds: > > cra> On Sun, Jun 05, 2005 at 10:35:13PM -0400, Andy Stewart wrote: >>>I've tried setting "nmi_watchdog=1" to see if I could get an "oops" when >>>it hard locks - no dice. Do you know any other tricks I could try to >>>see if it is the kernel which is locking up? I'm running SuSE's version >>>of 2.6.8. > > cra> Have you enabled the kernel Magic SysRq key and tried typing that when > cra> it locks up? > > cra> sysctl -w kernel.sysrq = 1 > > cra> When it locks up: > > cra> Alt-SysRq-? for help, then try dumping the PC, Tasks, etc. You may > cra> even be able to do an Emergency RO Mount and Sync before doing a Boot. > > This requires that you are not in X windows at the time of the lockup. See > my other email in this thead. HI all, The damn thing locked up on me *again* this evening, while trying to reply to this thread. On every occasion, I have been in X windows at the time of the lockup. I am either actively working or the screen saver is running, and the system locks up sometime in the middle of the night. What I can do is leave X windows running, but switch to a console before going to bed. If it hangs in the night, will I then be able to use the Alt-SysReq key sequences to get debug info? Thanks! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpinZHl0iXDssISsRAvwdAJ96My0XChSA5+kGI8XxzULjy27RMQCfVvey aSEYIUzjrxzfhr38eAs6q7Q= =+3IF -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 J Schonberg wrote: > > When the machine hung, did the num lock key still turn on and off the > led on the keyboard? If not, then other key sequences are probably > not working either. Is this a USB keyboard? > Hi Jason, I don't know if pressing the num lock key would cause the keyboard LED to toggle while the system is locked up. I'll try that next time the system locks. The keyboard is an old AT keyboard connected to my KVM switch, to which 3 other systems are connected via their keyboard/mouse connectors (not USB in any case). Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpidEHl0iXDssISsRAr5LAJwLAKrL84fdZw39XaYAcYwM8dTvUQCfQLDW lVQltv7fJj7tWFmpXC6l1es= =364+ -----END PGP SIGNATURE-----

June 7, 2005