- WLUG - wlug.mailman3.com

Re: [Wlug] HOWTO debug hard lockups

by Gary Hanley

On Tue, 7 Jun 2005, Andy Stewart wrote: > I'd rather not give up KDE, but I could try blackbox for a while to see > what happens. Well, my guess is that it is a hardware issue How about the MB settings, like the memory timings/clock ratios? Sometimes this can cause problems if they are set too aggressively. Usually there are one-off options in the BIOS menu for stable settings and speed settings so you don't have to tweak everything. Anything there you can check? Maybe down-shift your settings a little and see if the problem goes away. I haven't had the pleasure of working with a dual system every so I'm not familiar with any related BIOS settings you may need to modify. -- Gary

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Baris Hasdemir wrote: > On Tue, 2005-06-07 at 20:10 -0400, Andy Stewart wrote: >>-----BEGIN PGP SIGNED MESSAGE----- >>Hash: SHA1 >> >>Baris Hasdemir wrote: >>>On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: >>>>I'd rather not give up KDE, but I could try blackbox for a while to see >>>>what happens. >>>It might also worth to disable hardware acceleration support. For >>>example you can use nv instead of nvidia, for a while. >>> >>>baris >>HI Baris, >> >>Thanks for the tip, but my video card is an older one based on the ATI >>Radeon 7500. Not everything in this machine is new. >> >>Thanks, > > Are you using ati or radeon module? if using latter you have the > hardware acceleration support, if I recall correctly. > HI Baris, Hmmm, let me check. It appears that I'm using the radeon module. I'll make a note to try disabling hardware acceleration support for the video card. Thanks for the idea! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpl32Hl0iXDssISsRAuKKAJ9nkqIT8By9WV4h9htjaiy7Sb/KpACbBmwF jInvBZ/LKPzIlezoXl7Bc+k= =qs7r -----END PGP SIGNATURE-----

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Baris Hasdemir

On Tue, 2005-06-07 at 20:10 -0400, Andy Stewart wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Baris Hasdemir wrote: > > On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: > >>I'd rather not give up KDE, but I could try blackbox for a while to see > >>what happens. > > > > It might also worth to disable hardware acceleration support. For > > example you can use nv instead of nvidia, for a while. > > > > baris > > HI Baris, > > Thanks for the tip, but my video card is an older one based on the ATI > Radeon 7500. Not everything in this machine is new. > > Thanks, Are you using ati or radeon module? if using latter you have the hardware acceleration support, if I recall correctly. baris

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Walt Sawyer wrote: > Andy, > Are you sure it's the hard drive? Try running Knoppix live CD for a > while. If it's still running after days(max uptime)+5, then perhaps > it's something else. > > If this message is redundant, ignore it! > Walt HI Walt, I have suffered from hard drive problems on previous systems, and in each of those cases, I had strange errors reported in the /var/log/messages file. That's not to say that a hard drive problem couldn't cause a lockup, but that hasn't been my past experience. With the lockups I'm currently seeing, I'm not seeing any error messages in the /var/log/messages file. At this point, I think anything and everything is suspect until proven otherwise. Thanks for the ideas! I'm jotting them all down. Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpjjYHl0iXDssISsRAvQFAJ4w8Q4A5Ai25Se0pgkoKdYWo9AFcACdF3Q4 bC9BhLL4vESujGOMSsA4R2o= =+CpT -----END PGP SIGNATURE-----

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Baris Hasdemir wrote: > On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: >>I'd rather not give up KDE, but I could try blackbox for a while to see >>what happens. > > It might also worth to disable hardware acceleration support. For > example you can use nv instead of nvidia, for a while. > > baris HI Baris, Thanks for the tip, but my video card is an older one based on the ATI Radeon 7500. Not everything in this machine is new. Thanks, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpjdrHl0iXDssISsRAoz4AJoCqnNFec1qMaYhLluDuyajGurPGQCfYbSi H4FuxFHjw6+5FVAfcaoWLjY= =3+wG -----END PGP SIGNATURE-----

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Walt Sawyer

Andy, Are you sure it's the hard drive? Try running Knoppix live CD for a while. If it's still running after days(max uptime)+5, then perhaps it's something else. If this message is redundant, ignore it! Walt On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Jeff Moyer wrote: > > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: > > > > andystewart> HI gang, > > > > andystewart> My dual Opteron machine is not happy. I cannot get more than > > andystewart> 7 straight days of uptime without getting a hard lock, > > andystewart> requiring a reboot. (My definition of hard lock is: machine > > andystewart> responds neither to keyboard input, mouse input, nor network > > andystewart> pings). > > > > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had > > andystewart> 3 tonight, and 3-4 on a previous occasion while running > > andystewart> OpenOffice). It makes no sense to me that an application run > > andystewart> as a normal user could lockup a machine. > > > > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an > > andystewart> "oops" when it hard locks - no dice. Do you know any other > > andystewart> tricks I could try to see if it is the kernel which is locking > > andystewart> up? I'm running SuSE's version of 2.6.8. > > > > Did you verify that NMIs are being delivered? After boot, cat > > /proc/interrupts and make sure the NMI line is non-zero. Also note that, > > at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 > > for Opterons (i.e. you shouldn't need to manually set it). > > HI Jeff, > > Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me > think they are NOT being delivered. How would I go about solving *that* > little problem? I did a "cat /proc/cmdline" to insure that I had > "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've > been seeking. > > > If the NMI watchdog works, it will print a message to the console. > > However, you will not see this if you are in X windows. Do you have a > > serial console hooked up, by any chance? I strongly suggest it if you have > > the means. > > I think I have a cable to which I could connect the serial port of the > Opteron to the serial port of another Linux box (and then use minicom or > some such terminal program). > > > > > Aside from this, if it is indeed a hard lockup, there is really nothing you > > can do (without purchasing other hardware to help debug the problem). > > Yeah, I was afraid of that. > > > > > Please give these suggestions a shot and let us know how it goes. > > Shall do - thanks, everybody! > > Oh, be advised that when you smash your fist on the keyboard after your > system locks up for the umpteenth time, that a lot of dead skin cells > will come flying upward out of the bowels of the keyboard. I would > recommend safety glasses. > > Later, > > Andy > > - -- > Andy Stewart, Founder > Worcester Linux Users' Group > Worcester, MA, USA > http://www.wlug.org > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.5 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > > iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV > u6O8n+PSW4M0I1MHXsH06Xo= > =ViVg > -----END PGP SIGNATURE----- > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > --

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Baris Hasdemir

On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote: > I'd rather not give up KDE, but I could try blackbox for a while to see > what happens. It might also worth to disable hardware acceleration support. For example you can use nv instead of nvidia, for a while. baris

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote: > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: > > andystewart> HI gang, > > andystewart> My dual Opteron machine is not happy. I cannot get more than > andystewart> 7 straight days of uptime without getting a hard lock, > andystewart> requiring a reboot. (My definition of hard lock is: machine > andystewart> responds neither to keyboard input, mouse input, nor network > andystewart> pings). > > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had > andystewart> 3 tonight, and 3-4 on a previous occasion while running > andystewart> OpenOffice). It makes no sense to me that an application run > andystewart> as a normal user could lockup a machine. > > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an > andystewart> "oops" when it hard locks - no dice. Do you know any other > andystewart> tricks I could try to see if it is the kernel which is locking > andystewart> up? I'm running SuSE's version of 2.6.8. > > Did you verify that NMIs are being delivered? After boot, cat > /proc/interrupts and make sure the NMI line is non-zero. Also note that, > at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 > for Opterons (i.e. you shouldn't need to manually set it). HI Jeff, Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me think they are NOT being delivered. How would I go about solving *that* little problem? I did a "cat /proc/cmdline" to insure that I had "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've been seeking. > If the NMI watchdog works, it will print a message to the console. > However, you will not see this if you are in X windows. Do you have a > serial console hooked up, by any chance? I strongly suggest it if you have > the means. I think I have a cable to which I could connect the serial port of the Opteron to the serial port of another Linux box (and then use minicom or some such terminal program). > > Aside from this, if it is indeed a hard lockup, there is really nothing you > can do (without purchasing other hardware to help debug the problem). Yeah, I was afraid of that. > > Please give these suggestions a shot and let us know how it goes. Shall do - thanks, everybody! Oh, be advised that when you smash your fist on the keyboard after your system locks up for the umpteenth time, that a lot of dead skin cells will come flying upward out of the bowels of the keyboard. I would recommend safety glasses. Later, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV u6O8n+PSW4M0I1MHXsH06Xo= =ViVg -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Gary at home wrote: > Andy Stewart wrote: >>It makes no >>sense to me that an application run as a normal user could lockup a machine. > > Oh it's easy to crash a Linux machine with a fork-fest - any who can > fork a process can do it, so a rouge app can certainly do the same. > > What desktop are you running? I previously had similar problems with > Gnome on my Athlon XP under Gentoo that mysteriously stopped after a few > package updates...and I found no bug alerts for my symptoms at the time. > > Any chance you can run top or some other real-time system monitor when > the crash happens? > > -- Gary HI Gary, I am running KDE version 3.3.0 (SuSE 9.2). I could leave top running on a console to see if I can capture some useful info when the system locks up. I'll try that - thanks for the suggestion. I'd rather not give up KDE, but I could try blackbox for a while to see what happens. Thanks for the suggestions, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpipsHl0iXDssISsRAuPGAJ0T6JzfgqbGuWrB05uuJaPMohGytACeIbwJ mHYyFd2lDE8JbLQ4FAXC7Dk= =sdOC -----END PGP SIGNATURE-----

June 7, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote: > ==> Regarding Re: [Wlug] HOWTO debug hard lockups; "Chuck R. Anderson" <cra(a)WPI.EDU> adds: > > cra> On Sun, Jun 05, 2005 at 10:35:13PM -0400, Andy Stewart wrote: >>>I've tried setting "nmi_watchdog=1" to see if I could get an "oops" when >>>it hard locks - no dice. Do you know any other tricks I could try to >>>see if it is the kernel which is locking up? I'm running SuSE's version >>>of 2.6.8. > > cra> Have you enabled the kernel Magic SysRq key and tried typing that when > cra> it locks up? > > cra> sysctl -w kernel.sysrq = 1 > > cra> When it locks up: > > cra> Alt-SysRq-? for help, then try dumping the PC, Tasks, etc. You may > cra> even be able to do an Emergency RO Mount and Sync before doing a Boot. > > This requires that you are not in X windows at the time of the lockup. See > my other email in this thead. HI all, The damn thing locked up on me *again* this evening, while trying to reply to this thread. On every occasion, I have been in X windows at the time of the lockup. I am either actively working or the screen saver is running, and the system locks up sometime in the middle of the night. What I can do is leave X windows running, but switch to a console before going to bed. If it hangs in the night, will I then be able to use the Alt-SysReq key sequences to get debug info? Thanks! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpinZHl0iXDssISsRAvwdAJ96My0XChSA5+kGI8XxzULjy27RMQCfVvey aSEYIUzjrxzfhr38eAs6q7Q= =+3IF -----END PGP SIGNATURE-----

June 7, 2005