-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 HI gang, My dual Opteron machine is not happy. I cannot get more than 7 straight days of uptime without getting a hard lock, requiring a reboot. (My definition of hard lock is: machine responds neither to keyboard input, mouse input, nor network pings). I can stimulate hard locks by running OpenOffice 1.1.3 (I had 3 tonight, and 3-4 on a previous occasion while running OpenOffice). It makes no sense to me that an application run as a normal user could lockup a machine. I've tried setting "nmi_watchdog=1" to see if I could get an "oops" when it hard locks - no dice. Do you know any other tricks I could try to see if it is the kernel which is locking up? I'm running SuSE's version of 2.6.8. It may also be the hardware itself which is locking up. I have suspected that I might have "warm" hardware and have taken steps to better cool the machine. One CPU runs at a high of 63 degrees C, while the other one runs at 58 degrees C both while running Seti@Home (which heats the CPUs a significant 5-10 degrees C versus idle). The cooler CPU has a Zalman 7000 series copper heatsink/fan combo, while the warmer CPU has an undersized block of metal and a small fan (thanks to the MB manufacturer MSI). AMD quotes a max operating temperature of 70 degress C, so although quite hot, I think I'm still within spec, but I'm uncomfortably close. The temperature on the surface of the SCSI hard drive is 95 degrees F. The room temperature is a balmy 85 degrees (hey, its New England in the summer time!). The machine has passed memtest many times. If it is a heat problem, I'm royally screwed when the real summer weather arrives. I want to install a better heat sink on CPU 2 (hence my previous email about a heat sink retaining bracket). Physical tolerances on this MB are really tough to satisfy (MSI K8T Master2 FAR). The heat sink tooling on the MB is for socket 478 even though the MB uses AMD opteron 244s (yes, this is quite bizarre, er, non-standard). If anybody has any ideas which might help, I'm all ears (as it were). Thanks! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCo7ZhHl0iXDssISsRAoGWAJ4ttqZS2SSgdlGwdnSp6LNbzUmLaQCfbktB KWUM+7EQAu3jIIKbmdnDkZk= =aI/l -----END PGP SIGNATURE-----
On Sun, Jun 05, 2005 at 10:35:13PM -0400, Andy Stewart wrote:
I've tried setting "nmi_watchdog=1" to see if I could get an "oops" when it hard locks - no dice. Do you know any other tricks I could try to see if it is the kernel which is locking up? I'm running SuSE's version of 2.6.8.
Have you enabled the kernel Magic SysRq key and tried typing that when it locks up? sysctl -w kernel.sysrq = 1 When it locks up: Alt-SysRq-? for help, then try dumping the PC, Tasks, etc. You may even be able to do an Emergency RO Mount and Sync before doing a Boot.
==> Regarding Re: [Wlug] HOWTO debug hard lockups; "Chuck R. Anderson" <cra@WPI.EDU> adds: cra> On Sun, Jun 05, 2005 at 10:35:13PM -0400, Andy Stewart wrote:
I've tried setting "nmi_watchdog=1" to see if I could get an "oops" when it hard locks - no dice. Do you know any other tricks I could try to see if it is the kernel which is locking up? I'm running SuSE's version of 2.6.8.
cra> Have you enabled the kernel Magic SysRq key and tried typing that when cra> it locks up? cra> sysctl -w kernel.sysrq = 1 cra> When it locks up: cra> Alt-SysRq-? for help, then try dumping the PC, Tasks, etc. You may cra> even be able to do an Emergency RO Mount and Sync before doing a Boot. This requires that you are not in X windows at the time of the lockup. See my other email in this thead. Also note that, due to no specific instances or facts to back up my statement, I don't like doing the 'Sync'. If the system is locked up, who is to say that you are going to write out valid data to the disk? I'd rather just reboot and let the journal recover things. -Jeff
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote:
==> Regarding Re: [Wlug] HOWTO debug hard lockups; "Chuck R. Anderson" <cra@WPI.EDU> adds:
cra> On Sun, Jun 05, 2005 at 10:35:13PM -0400, Andy Stewart wrote:
I've tried setting "nmi_watchdog=1" to see if I could get an "oops" when it hard locks - no dice. Do you know any other tricks I could try to see if it is the kernel which is locking up? I'm running SuSE's version of 2.6.8.
cra> Have you enabled the kernel Magic SysRq key and tried typing that when cra> it locks up?
cra> sysctl -w kernel.sysrq = 1
cra> When it locks up:
cra> Alt-SysRq-? for help, then try dumping the PC, Tasks, etc. You may cra> even be able to do an Emergency RO Mount and Sync before doing a Boot.
This requires that you are not in X windows at the time of the lockup. See my other email in this thead.
HI all, The damn thing locked up on me *again* this evening, while trying to reply to this thread. On every occasion, I have been in X windows at the time of the lockup. I am either actively working or the screen saver is running, and the system locks up sometime in the middle of the night. What I can do is leave X windows running, but switch to a console before going to bed. If it hangs in the night, will I then be able to use the Alt-SysReq key sequences to get debug info? Thanks! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpinZHl0iXDssISsRAvwdAJ96My0XChSA5+kGI8XxzULjy27RMQCfVvey aSEYIUzjrxzfhr38eAs6q7Q= =+3IF -----END PGP SIGNATURE-----
Andy, The best thing to do here is to run a serial console to another one of your systems and use THAT to try and catch the issue. You can then do a SysRq from the remote serial console and/or catch any error messages. Also, are there any messages in dmesg or /var/log/messages? How does the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see if the problem repeats. John
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 John Stoffel wrote:
Andy,
The best thing to do here is to run a serial console to another one of your systems and use THAT to try and catch the issue. You can then do a SysRq from the remote serial console and/or catch any error messages.
Also, are there any messages in dmesg or /var/log/messages? How does the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see if the problem repeats.
John
HI John, I'm going to give the serial console a try - I may have a cable lying around. I upgraded to the 2.6.11.11 kernel which did have several fixes for K8. We'll see how that goes. Thanks for the ideas! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCp2yhHl0iXDssISsRAqxcAJwLntOh9GpNmefW/+JdwbAUMRJKBACfbV1P s2eqMUnY6rjZaIfI6MzZLqQ= =aLfl -----END PGP SIGNATURE-----
Andy Stewart wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
John Stoffel wrote:
Andy,
The best thing to do here is to run a serial console to another one of your systems and use THAT to try and catch the issue. You can then do a SysRq from the remote serial console and/or catch any error messages.
Also, are there any messages in dmesg or /var/log/messages? How does the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see if the problem repeats.
John
HI John,
I'm going to give the serial console a try - I may have a cable lying around.
I upgraded to the 2.6.11.11 kernel which did have several fixes for K8. We'll see how that goes.
Thanks for the ideas!
Andy
- -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFCp2yhHl0iXDssISsRAqxcAJwLntOh9GpNmefW/+JdwbAUMRJKBACfbV1P s2eqMUnY6rjZaIfI6MzZLqQ= =aLfl -----END PGP SIGNATURE----- _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
Andy, If that doesn't help try using the system without the kvm and see if you get the same behavior. I have seen odd things in the past when using a faulty kvm switch. Mike
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chuck R. Anderson wrote:
On Sun, Jun 05, 2005 at 10:35:13PM -0400, Andy Stewart wrote:
I've tried setting "nmi_watchdog=1" to see if I could get an "oops" when it hard locks - no dice. Do you know any other tricks I could try to see if it is the kernel which is locking up? I'm running SuSE's version of 2.6.8.
Have you enabled the kernel Magic SysRq key and tried typing that when it locks up?
sysctl -w kernel.sysrq = 1
When it locks up:
Alt-SysRq-? for help, then try dumping the PC, Tasks, etc. You may even be able to do an Emergency RO Mount and Sync before doing a Boot. _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
HI Chuck, That's a good idea which I've implemented per your suggestion. Now, I just have to wait for the next hang (hey, I could update the WLUG website with OpenOffice!). I'll keep you posted. Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpMefHl0iXDssISsRAnEsAJ9bcbaboREGjDKodlg4QS/I3+UB1ACfZF6x yGYCi38pKMmFAUyrE89aEfA= =r3Kq -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Chuck et al, The machine just locked up again (and it is much cooler today than yesterday). I wasn't running OpenOffice this time. I tried alt-printscreen-? and nothing happened. I tried alt-printscreen-anything and nothing happened. crl-alt-delete, ctrl-alt-backspace - no keyboard input at all was recognized as far as I can tell. Also, I found a lot of entries about this MB on the MSI Forum and it seems that the temp sensors register about 15% higher that reality. gkrellm allows me to tweak the readings, so I did. An independent temperature measurement confirms that the onboard sensors are registering about 15% high, and that I'm not really frying my CPUs as I had thought. At idle, my CPUs are around 43 degrees C. This makes this issue all the more puzzling. Other thoughts? I'm about to fire up an all night run of memtest. Thanks, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpQp+Hl0iXDssISsRAnBYAJ9WZDDB0G/y6lL31aYJDYPfNHJe1QCdHkSf k1E6p2ayY5rGt/+Lamb0VzU= =w7NU -----END PGP SIGNATURE-----
--- Andy Stewart <andystewart@comcast.net> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Chuck et al,
The machine just locked up again (and it is much cooler today than yesterday). I wasn't running OpenOffice this time. I tried alt-printscreen-? and nothing happened. I tried alt-printscreen-anything and nothing happened. crl-alt-delete, ctrl-alt-backspace - no keyboard input at all was recognized as far as I can tell.
When the machine hung, did the num lock key still turn on and off the led on the keyboard? If not, then other key sequences are probably not working either. Is this a USB keyboard? Jason. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 J Schonberg wrote:
When the machine hung, did the num lock key still turn on and off the led on the keyboard? If not, then other key sequences are probably not working either. Is this a USB keyboard?
Hi Jason, I don't know if pressing the num lock key would cause the keyboard LED to toggle while the system is locked up. I'll try that next time the system locks. The keyboard is an old AT keyboard connected to my KVM switch, to which 3 other systems are connected via their keyboard/mouse connectors (not USB in any case). Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpidEHl0iXDssISsRAr5LAJwLAKrL84fdZw39XaYAcYwM8dTvUQCfQLDW lVQltv7fJj7tWFmpXC6l1es= =364+ -----END PGP SIGNATURE-----
==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds: andystewart> HI gang, andystewart> My dual Opteron machine is not happy. I cannot get more than andystewart> 7 straight days of uptime without getting a hard lock, andystewart> requiring a reboot. (My definition of hard lock is: machine andystewart> responds neither to keyboard input, mouse input, nor network andystewart> pings). andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had andystewart> 3 tonight, and 3-4 on a previous occasion while running andystewart> OpenOffice). It makes no sense to me that an application run andystewart> as a normal user could lockup a machine. andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an andystewart> "oops" when it hard locks - no dice. Do you know any other andystewart> tricks I could try to see if it is the kernel which is locking andystewart> up? I'm running SuSE's version of 2.6.8. Did you verify that NMIs are being delivered? After boot, cat /proc/interrupts and make sure the NMI line is non-zero. Also note that, at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 for Opterons (i.e. you shouldn't need to manually set it). If the NMI watchdog works, it will print a message to the console. However, you will not see this if you are in X windows. Do you have a serial console hooked up, by any chance? I strongly suggest it if you have the means. Aside from this, if it is indeed a hard lockup, there is really nothing you can do (without purchasing other hardware to help debug the problem). Please give these suggestions a shot and let us know how it goes. Thanks! -Jeff
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote:
==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds:
andystewart> HI gang,
andystewart> My dual Opteron machine is not happy. I cannot get more than andystewart> 7 straight days of uptime without getting a hard lock, andystewart> requiring a reboot. (My definition of hard lock is: machine andystewart> responds neither to keyboard input, mouse input, nor network andystewart> pings).
andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had andystewart> 3 tonight, and 3-4 on a previous occasion while running andystewart> OpenOffice). It makes no sense to me that an application run andystewart> as a normal user could lockup a machine.
andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an andystewart> "oops" when it hard locks - no dice. Do you know any other andystewart> tricks I could try to see if it is the kernel which is locking andystewart> up? I'm running SuSE's version of 2.6.8.
Did you verify that NMIs are being delivered? After boot, cat /proc/interrupts and make sure the NMI line is non-zero. Also note that, at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 for Opterons (i.e. you shouldn't need to manually set it).
HI Jeff, Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me think they are NOT being delivered. How would I go about solving *that* little problem? I did a "cat /proc/cmdline" to insure that I had "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've been seeking.
If the NMI watchdog works, it will print a message to the console. However, you will not see this if you are in X windows. Do you have a serial console hooked up, by any chance? I strongly suggest it if you have the means.
I think I have a cable to which I could connect the serial port of the Opteron to the serial port of another Linux box (and then use minicom or some such terminal program).
Aside from this, if it is indeed a hard lockup, there is really nothing you can do (without purchasing other hardware to help debug the problem).
Yeah, I was afraid of that.
Please give these suggestions a shot and let us know how it goes.
Shall do - thanks, everybody! Oh, be advised that when you smash your fist on the keyboard after your system locks up for the umpteenth time, that a lot of dead skin cells will come flying upward out of the bowels of the keyboard. I would recommend safety glasses. Later, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV u6O8n+PSW4M0I1MHXsH06Xo= =ViVg -----END PGP SIGNATURE-----
Andy, Are you sure it's the hard drive? Try running Knoppix live CD for a while. If it's still running after days(max uptime)+5, then perhaps it's something else. If this message is redundant, ignore it! Walt On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jeff Moyer wrote:
==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds:
andystewart> HI gang,
andystewart> My dual Opteron machine is not happy. I cannot get more than andystewart> 7 straight days of uptime without getting a hard lock, andystewart> requiring a reboot. (My definition of hard lock is: machine andystewart> responds neither to keyboard input, mouse input, nor network andystewart> pings).
andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had andystewart> 3 tonight, and 3-4 on a previous occasion while running andystewart> OpenOffice). It makes no sense to me that an application run andystewart> as a normal user could lockup a machine.
andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an andystewart> "oops" when it hard locks - no dice. Do you know any other andystewart> tricks I could try to see if it is the kernel which is locking andystewart> up? I'm running SuSE's version of 2.6.8.
Did you verify that NMIs are being delivered? After boot, cat /proc/interrupts and make sure the NMI line is non-zero. Also note that, at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 for Opterons (i.e. you shouldn't need to manually set it).
HI Jeff,
Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me think they are NOT being delivered. How would I go about solving *that* little problem? I did a "cat /proc/cmdline" to insure that I had "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've been seeking.
If the NMI watchdog works, it will print a message to the console. However, you will not see this if you are in X windows. Do you have a serial console hooked up, by any chance? I strongly suggest it if you have the means.
I think I have a cable to which I could connect the serial port of the Opteron to the serial port of another Linux box (and then use minicom or some such terminal program).
Aside from this, if it is indeed a hard lockup, there is really nothing you can do (without purchasing other hardware to help debug the problem).
Yeah, I was afraid of that.
Please give these suggestions a shot and let us know how it goes.
Shall do - thanks, everybody!
Oh, be advised that when you smash your fist on the keyboard after your system locks up for the umpteenth time, that a lot of dead skin cells will come flying upward out of the bowels of the keyboard. I would recommend safety glasses.
Later,
Andy
- -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV u6O8n+PSW4M0I1MHXsH06Xo= =ViVg -----END PGP SIGNATURE----- _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
--
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Walt Sawyer wrote:
Andy, Are you sure it's the hard drive? Try running Knoppix live CD for a while. If it's still running after days(max uptime)+5, then perhaps it's something else.
If this message is redundant, ignore it! Walt
HI Walt, I have suffered from hard drive problems on previous systems, and in each of those cases, I had strange errors reported in the /var/log/messages file. That's not to say that a hard drive problem couldn't cause a lockup, but that hasn't been my past experience. With the lockups I'm currently seeing, I'm not seeing any error messages in the /var/log/messages file. At this point, I think anything and everything is suspect until proven otherwise. Thanks for the ideas! I'm jotting them all down. Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpjjYHl0iXDssISsRAvQFAJ4w8Q4A5Ai25Se0pgkoKdYWo9AFcACdF3Q4 bC9BhLL4vESujGOMSsA4R2o= =+CpT -----END PGP SIGNATURE-----
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Walt Sawyer <wsawyer@norfolk-county.com> adds: wsawyer> Andy, Are you sure it's the hard drive? Try running Knoppix live wsawyer> CD for a while. If it's still running after days(max uptime)+5, wsawyer> then perhaps it's something else. This changes things entirely. He'll be running a different kernel, so this test is not useful, in my opinion. -Jeff wsawyer> If this message is redundant, ignore it! Walt wsawyer> On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jeff Moyer wrote: > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds:
andystewart> HI gang,
andystewart> My dual Opteron machine is not happy. I cannot get more
than > andystewart> 7 straight days of uptime without getting a hard lock, > andystewart> requiring a reboot. (My definition of hard lock is: machine > andystewart> responds neither to keyboard input, mouse input, nor network > andystewart> pings).
andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I
had > andystewart> 3 tonight, and 3-4 on a previous occasion while running > andystewart> OpenOffice). It makes no sense to me that an application run > andystewart> as a normal user could lockup a machine.
andystewart> I've tried setting "nmi_watchdog=1" to see if I could get
an > andystewart> "oops" when it hard locks - no dice. Do you know any other > andystewart> tricks I could try to see if it is the kernel which is locking > andystewart> up? I'm running SuSE's version of 2.6.8.
Did you verify that NMIs are being delivered? After boot, cat >
/proc/interrupts and make sure the NMI line is non-zero. Also note that, > at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 > for Opterons (i.e. you shouldn't need to manually set it).
HI Jeff,
Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me think they are NOT being delivered. How would I go about solving *that* little problem? I did a "cat /proc/cmdline" to insure that I had "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've been seeking.
If the NMI watchdog works, it will print a message to the console. > However, you will not see this if you are in X windows. Do you have a > serial console hooked up, by any chance? I strongly suggest it if you have > the means.
I think I have a cable to which I could connect the serial port of the Opteron to the serial port of another Linux box (and then use minicom or some such terminal program).
Aside from this, if it is indeed a hard lockup, there is really
nothing you > can do (without purchasing other hardware to help debug the problem).
Yeah, I was afraid of that.
Please give these suggestions a shot and let us know how it goes.
Shall do - thanks, everybody!
Oh, be advised that when you smash your fist on the keyboard after your system locks up for the umpteenth time, that a lot of dead skin cells will come flying upward out of the bowels of the keyboard. I would recommend safety glasses.
Later,
Andy
- -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV u6O8n+PSW4M0I1MHXsH06Xo= =ViVg -----END PGP SIGNATURE----- _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
wsawyer> -- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN"> wsawyer> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; wsawyer> CHARSET=UTF-8"> <META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2"> wsawyer> </HEAD> <BODY> Andy,<BR> Are you sure it's the hard drive? wsawyer> Try running Knoppix live CD for a while. If it's still wsawyer> running after days(max uptime)+5, then perhaps it's something wsawyer> else.<BR> <BR> If this message is redundant, ignore it!<BR> wsawyer> Walt<BR> <BR> On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wsawyer> wrote: <BLOCKQUOTE TYPE=CITE> <PRE> <FONT wsawyer> COLOR="#000000">-----BEGIN PGP SIGNED MESSAGE-----</FONT> <FONT wsawyer> COLOR="#000000">Hash: SHA1</FONT> wsawyer> <FONT COLOR="#000000">Jeff Moyer wrote:</FONT> <FONT wsawyer> COLOR="#000000">> ==> Regarding [Wlug] HOWTO debug hard wsawyer> lockups; Andy Stewart <<A wsawyer> HREF="mailto:andystewart@comcast.net">andystewart@comcast.net</A>> wsawyer> adds:</FONT> <FONT COLOR="#000000">> </FONT> <FONT wsawyer> COLOR="#000000">> andystewart> HI gang,</FONT> <FONT wsawyer> COLOR="#000000">> </FONT> <FONT COLOR="#000000">> wsawyer> andystewart> My dual Opteron machine is not happy. I cannot wsawyer> get more than</FONT> <FONT COLOR="#000000">> andystewart> 7 wsawyer> straight days of uptime without getting a hard lock,</FONT> <FONT wsawyer> COLOR="#000000">> andystewart> requiring a reboot. (My wsawyer> definition of hard lock is: machine</FONT> <FONT wsawyer> COLOR="#000000">> andystewart> responds neither to keyboard wsawyer> input, mouse input, nor network</FONT> <FONT COLOR="#000000">> wsawyer> andystewart> pings).</FONT> <FONT COLOR="#000000">> </FONT> wsawyer> <FONT COLOR="#000000">> andystewart> I can stimulate hard wsawyer> locks by running OpenOffice 1.1.3 (I had</FONT> <FONT wsawyer> COLOR="#000000">> andystewart> 3 tonight, and 3-4 on a wsawyer> previous occasion while running</FONT> <FONT COLOR="#000000">> wsawyer> andystewart> OpenOffice). It makes no sense to me that an wsawyer> application run</FONT> <FONT COLOR="#000000">> andystewart> wsawyer> as a normal user could lockup a machine.</FONT> <FONT wsawyer> COLOR="#000000">> </FONT> <FONT COLOR="#000000">> wsawyer> andystewart> I've tried setting "nmi_watchdog=1" to wsawyer> see if I could get an</FONT> <FONT COLOR="#000000">> wsawyer> andystewart> "oops" when it hard locks - no dice. Do wsawyer> you know any other</FONT> <FONT COLOR="#000000">> wsawyer> andystewart> tricks I could try to see if it is the kernel wsawyer> which is locking</FONT> <FONT COLOR="#000000">> andystewart> wsawyer> up? I'm running SuSE's version of 2.6.8.</FONT> <FONT wsawyer> COLOR="#000000">> </FONT> <FONT COLOR="#000000">> Did you wsawyer> verify that NMIs are being delivered? After boot, cat</FONT> wsawyer> <FONT COLOR="#000000">> /proc/interrupts and make sure the NMI wsawyer> line is non-zero. Also note that,</FONT> <FONT wsawyer> COLOR="#000000">> at least with upstream and Red Hat kernels, wsawyer> the nmi_watchdog defaults to 1</FONT> <FONT COLOR="#000000">> wsawyer> for Opterons (i.e. you shouldn't need to manually set it).</FONT> wsawyer> <FONT COLOR="#000000">HI Jeff,</FONT> wsawyer> <FONT COLOR="#000000">Well, this is weird, I'm seeing a ZERO count wsawyer> for NMIs, so that makes me</FONT> <FONT COLOR="#000000">think they wsawyer> are NOT being delivered. How would I go about solving wsawyer> *that*</FONT> <FONT COLOR="#000000">little problem? I did a wsawyer> "cat /proc/cmdline" to insure that I had</FONT> <FONT wsawyer> COLOR="#000000">"nmi_watchdog=1" and indeed it is there. wsawyer> Perhaps this is the clue we've</FONT> <FONT COLOR="#000000">been wsawyer> seeking.</FONT> wsawyer> <FONT COLOR="#000000">> If the NMI watchdog works, it will wsawyer> print a message to the console.</FONT> <FONT COLOR="#000000">> wsawyer> However, you will not see this if you are in X windows. Do you wsawyer> have a</FONT> <FONT COLOR="#000000">> serial console hooked up, wsawyer> by any chance? I strongly suggest it if you have</FONT> <FONT wsawyer> COLOR="#000000">> the means.</FONT> wsawyer> <FONT COLOR="#000000">I think I have a cable to which I could wsawyer> connect the serial port of the</FONT> <FONT wsawyer> COLOR="#000000">Opteron to the serial port of another Linux box wsawyer> (and then use minicom or</FONT> <FONT COLOR="#000000">some such wsawyer> terminal program).</FONT> wsawyer> <FONT COLOR="#000000">> </FONT> <FONT COLOR="#000000">> wsawyer> Aside from this, if it is indeed a hard lockup, there is really wsawyer> nothing you</FONT> <FONT COLOR="#000000">> can do (without wsawyer> purchasing other hardware to help debug the problem).</FONT> wsawyer> <FONT COLOR="#000000">Yeah, I was afraid of that.</FONT> wsawyer> <FONT COLOR="#000000">> </FONT> <FONT COLOR="#000000">> wsawyer> Please give these suggestions a shot and let us know how it wsawyer> goes.</FONT> wsawyer> <FONT COLOR="#000000">Shall do - thanks, everybody!</FONT> wsawyer> <FONT COLOR="#000000">Oh, be advised that when you smash your fist wsawyer> on the keyboard after your</FONT> <FONT COLOR="#000000">system wsawyer> locks up for the umpteenth time, that a lot of dead skin wsawyer> cells</FONT> <FONT COLOR="#000000">will come flying upward out of wsawyer> the bowels of the keyboard. I would</FONT> <FONT wsawyer> COLOR="#000000">recommend safety glasses.</FONT> wsawyer> <FONT COLOR="#000000">Later,</FONT> wsawyer> <FONT COLOR="#000000">Andy</FONT> wsawyer> <FONT COLOR="#000000">- --</FONT> <FONT COLOR="#000000">Andy wsawyer> Stewart, Founder</FONT> <FONT COLOR="#000000">Worcester Linux wsawyer> Users' Group</FONT> <FONT COLOR="#000000">Worcester, MA, wsawyer> USA</FONT> <FONT COLOR="#000000"><A wsawyer> HREF="http://www.wlug.org">http://www.wlug.org</A></FONT> wsawyer> <FONT COLOR="#000000">-----BEGIN PGP SIGNATURE-----</FONT> <FONT wsawyer> COLOR="#000000">Version: GnuPG v1.2.5 (GNU/Linux)</FONT> <FONT wsawyer> COLOR="#000000">Comment: Using GnuPG with Thunderbird - <A wsawyer> HREF="http://enigmail.mozdev.org">http://enigmail.mozdev.org</A></FONT> wsawyer> <FONT wsawyer> COLOR="#000000">iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV</FONT> wsawyer> <FONT COLOR="#000000">u6O8n+PSW4M0I1MHXsH06Xo=</FONT> <FONT wsawyer> COLOR="#000000">=ViVg</FONT> <FONT COLOR="#000000">-----END PGP wsawyer> SIGNATURE-----</FONT> <FONT wsawyer> COLOR="#000000">_______________________________________________</FONT> wsawyer> <FONT COLOR="#000000">Wlug mailing list</FONT> <FONT wsawyer> COLOR="#000000"><A wsawyer> HREF="mailto:Wlug@mail.wlug.org">Wlug@mail.wlug.org</A></FONT> wsawyer> <FONT COLOR="#000000"><A wsawyer> HREF="http://mail.wlug.org/mailman/listinfo/wlug">http://mail.wlug.org/mailman/listinfo/wlug</A></FONT> wsawyer> </PRE> </BLOCKQUOTE> <TABLE CELLSPACING="0" CELLPADDING="0" wsawyer> WIDTH="100%"> <TR> <TD> <PRE> -- </PRE> </TD> </TR> </TABLE> wsawyer> </BODY> </HTML> _______________________________________________ wsawyer> Wlug mailing list Wlug@mail.wlug.org wsawyer> http://mail.wlug.org/mailman/listinfo/wlug
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds: andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 andystewart> Jeff Moyer wrote:
==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds:
andystewart> HI gang,
andystewart> My dual Opteron machine is not happy. I cannot get more than andystewart> 7 straight days of uptime without getting a hard lock, andystewart> requiring a reboot. (My definition of hard lock is: machine andystewart> responds neither to keyboard input, mouse input, nor network andystewart> pings).
andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had andystewart> 3 tonight, and 3-4 on a previous occasion while running andystewart> OpenOffice). It makes no sense to me that an application run andystewart> as a normal user could lockup a machine.
andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an andystewart> "oops" when it hard locks - no dice. Do you know any other andystewart> tricks I could try to see if it is the kernel which is locking andystewart> up? I'm running SuSE's version of 2.6.8.
Did you verify that NMIs are being delivered? After boot, cat /proc/interrupts and make sure the NMI line is non-zero. Also note that, at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 for Opterons (i.e. you shouldn't need to manually set it).
andystewart> HI Jeff, andystewart> Well, this is weird, I'm seeing a ZERO count for NMIs, so that andystewart> makes me think they are NOT being delivered. How would I go andystewart> about solving *that* little problem? I did a "cat Well, you can try booting with nmi_watchdog=2. This will try to use the local APIC to deliver nmi's, but I haven't actually seen a dual processor system that required this (all of them I've seen work with nmi_watchdog=1). It is worth a try, however. andystewart> /proc/cmdline" to insure that I had "nmi_watchdog=1" and andystewart> indeed it is there. Perhaps this is the clue we've been andystewart> seeking. Well, it only tells you that the nmi_watchdog won't trigger. We still have no insight into what the problem might actually be.
If the NMI watchdog works, it will print a message to the console. However, you will not see this if you are in X windows. Do you have a serial console hooked up, by any chance? I strongly suggest it if you have the means.
andystewart> I think I have a cable to which I could connect the serial andystewart> port of the Opteron to the serial port of another Linux box andystewart> (and then use minicom or some such terminal program).
Aside from this, if it is indeed a hard lockup, there is really nothing you can do (without purchasing other hardware to help debug the problem).
andystewart> Yeah, I was afraid of that.
Please give these suggestions a shot and let us know how it goes.
andystewart> Shall do - thanks, everybody! andystewart> Oh, be advised that when you smash your fist on the keyboard andystewart> after your system locks up for the umpteenth time, that a lot andystewart> of dead skin cells will come flying upward out of the bowels andystewart> of the keyboard. I would recommend safety glasses. Noted. =) -Jeff
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote:
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds:
andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
andystewart> Jeff Moyer wrote:
==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds:
andystewart> HI gang, andystewart> My dual Opteron machine is not happy. I cannot get more than andystewart> 7 straight days of uptime without getting a hard lock, andystewart> requiring a reboot. (My definition of hard lock is: machine andystewart> responds neither to keyboard input, mouse input, nor network andystewart> pings). andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had andystewart> 3 tonight, and 3-4 on a previous occasion while running andystewart> OpenOffice). It makes no sense to me that an application run andystewart> as a normal user could lockup a machine. andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an andystewart> "oops" when it hard locks - no dice. Do you know any other andystewart> tricks I could try to see if it is the kernel which is locking andystewart> up? I'm running SuSE's version of 2.6.8.
Did you verify that NMIs are being delivered? After boot, cat /proc/interrupts and make sure the NMI line is non-zero. Also note that, at least with upstream and Red Hat kernels, the nmi_watchdog defaults to 1 for Opterons (i.e. you shouldn't need to manually set it).
andystewart> HI Jeff,
andystewart> Well, this is weird, I'm seeing a ZERO count for NMIs, so that andystewart> makes me think they are NOT being delivered. How would I go andystewart> about solving *that* little problem? I did a "cat
Well, you can try booting with nmi_watchdog=2. This will try to use the local APIC to deliver nmi's, but I haven't actually seen a dual processor system that required this (all of them I've seen work with nmi_watchdog=1). It is worth a try, however.
With nmi_watchdog=1, I see the following output from "dmesg | grep -i nmi" with the 2.6.11.11 kernel I very recently installed: Bootdata ok (command line is root=/dev/sda3 vga=0x31a selinux=0 console=tty0 resume=/dev/sda2 desktop elevator=as nmi_watchdog=1 splash=verbose) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) Kernel command line: root=/dev/sda3 vga=0x31a selinux=0 console=tty0 resume=/dev/sda2 desktop elevator=as nmi_watchdog=1 splash=verbose activating NMI Watchdog ... done. testing NMI watchdog ... CPU#1: NMI appears to be stuck (0)! I'll try nmi_watchdog=2 shortly. Thanks, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCp3v8Hl0iXDssISsRAoDJAJ0TonNTSEAd+4W0GRPwTW/UoGFssACdEEq9 8IgEFJ7VAL/FVeyPXrEapHg= =jcjE -----END PGP SIGNATURE-----
Andy Stewart wrote:
It makes no sense to me that an application run as a normal user could lockup a machine.
Oh it's easy to crash a Linux machine with a fork-fest - any who can fork a process can do it, so a rouge app can certainly do the same. What desktop are you running? I previously had similar problems with Gnome on my Athlon XP under Gentoo that mysteriously stopped after a few package updates...and I found no bug alerts for my symptoms at the time. Any chance you can run top or some other real-time system monitor when the crash happens? -- Gary
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Gary at home wrote:
Andy Stewart wrote:
It makes no sense to me that an application run as a normal user could lockup a machine.
Oh it's easy to crash a Linux machine with a fork-fest - any who can fork a process can do it, so a rouge app can certainly do the same.
What desktop are you running? I previously had similar problems with Gnome on my Athlon XP under Gentoo that mysteriously stopped after a few package updates...and I found no bug alerts for my symptoms at the time.
Any chance you can run top or some other real-time system monitor when the crash happens?
-- Gary
HI Gary, I am running KDE version 3.3.0 (SuSE 9.2). I could leave top running on a console to see if I can capture some useful info when the system locks up. I'll try that - thanks for the suggestion. I'd rather not give up KDE, but I could try blackbox for a while to see what happens. Thanks for the suggestions, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpipsHl0iXDssISsRAuPGAJ0T6JzfgqbGuWrB05uuJaPMohGytACeIbwJ mHYyFd2lDE8JbLQ4FAXC7Dk= =sdOC -----END PGP SIGNATURE-----
On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote:
I'd rather not give up KDE, but I could try blackbox for a while to see what happens.
It might also worth to disable hardware acceleration support. For example you can use nv instead of nvidia, for a while. baris
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Baris Hasdemir wrote:
On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote:
I'd rather not give up KDE, but I could try blackbox for a while to see what happens.
It might also worth to disable hardware acceleration support. For example you can use nv instead of nvidia, for a while.
baris
HI Baris, Thanks for the tip, but my video card is an older one based on the ATI Radeon 7500. Not everything in this machine is new. Thanks, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpjdrHl0iXDssISsRAoz4AJoCqnNFec1qMaYhLluDuyajGurPGQCfYbSi H4FuxFHjw6+5FVAfcaoWLjY= =3+wG -----END PGP SIGNATURE-----
On Tue, 2005-06-07 at 20:10 -0400, Andy Stewart wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Baris Hasdemir wrote:
On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote:
I'd rather not give up KDE, but I could try blackbox for a while to see what happens.
It might also worth to disable hardware acceleration support. For example you can use nv instead of nvidia, for a while.
baris
HI Baris,
Thanks for the tip, but my video card is an older one based on the ATI Radeon 7500. Not everything in this machine is new.
Thanks,
Are you using ati or radeon module? if using latter you have the hardware acceleration support, if I recall correctly. baris
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Baris Hasdemir wrote:
On Tue, 2005-06-07 at 20:10 -0400, Andy Stewart wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Baris Hasdemir wrote:
On Tue, 2005-06-07 at 19:14 -0400, Andy Stewart wrote:
I'd rather not give up KDE, but I could try blackbox for a while to see what happens. It might also worth to disable hardware acceleration support. For example you can use nv instead of nvidia, for a while.
baris HI Baris,
Thanks for the tip, but my video card is an older one based on the ATI Radeon 7500. Not everything in this machine is new.
Thanks,
Are you using ati or radeon module? if using latter you have the hardware acceleration support, if I recall correctly.
HI Baris, Hmmm, let me check. It appears that I'm using the radeon module. I'll make a note to try disabling hardware acceleration support for the video card. Thanks for the idea! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpl32Hl0iXDssISsRAuKKAJ9nkqIT8By9WV4h9htjaiy7Sb/KpACbBmwF jInvBZ/LKPzIlezoXl7Bc+k= =qs7r -----END PGP SIGNATURE-----
On Tue, 7 Jun 2005, Andy Stewart wrote:
I'd rather not give up KDE, but I could try blackbox for a while to see what happens.
Well, my guess is that it is a hardware issue How about the MB settings, like the memory timings/clock ratios? Sometimes this can cause problems if they are set too aggressively. Usually there are one-off options in the BIOS menu for stable settings and speed settings so you don't have to tweak everything. Anything there you can check? Maybe down-shift your settings a little and see if the problem goes away. I haven't had the pleasure of working with a dual system every so I'm not familiar with any related BIOS settings you may need to modify. -- Gary
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 HI gang, I have another clue - I found a way to lockup the system *guaranteed* within about 3 minutes. Go to this URL and play this game. It is mouse only. http://www.whtmtnliving.net/games/sports/beer_golf.htm I disconnected my keyboard and mouse from the KVM switch and connected them directly to my system - and it *still* locks up reliably when I play the golf game (above) with the 2.6.11.11 kernel. Note I'm also running X.org 6.8.1. Could this be some flakiness in my XF86Config file with the mouse protocol? I could isolate this further if I could find a way to lock up the system in console mode using gpm. Its time to go to CompUSA and spend $4.38 on that serial console cable... Later, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCsiV8Hl0iXDssISsRAoutAJ4qauOItdJnPoSXS2MuP149kKuMSgCdEXZ8 xP/T9Okh2ngQRZqZresZCDg= =2tw5 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andy Stewart wrote:
HI gang,
I have another clue - I found a way to lockup the system *guaranteed* within about 3 minutes. Go to this URL and play this game. It is mouse only.
OK, here's more to the saga of the machine suffering from hard lockups. Here's what I tried, and in all cases, the system still locks up playing beer_golf (but I am getting really good at this game!). This is all based on suggestions from all of you - thank you!! 1) Turned off video 3D acceleration. No effect, so I turned it back on. 2) Connected the KB and mouse directly to system (no KVM), no effect, so now the KVM is back in the picture. 3) Turned off sound hardware in the BIOS - no effect, so I turned it back on. 4) Tried a different video card (and this one was PCI, the original video card is AGP) - same deal, no effect, put back the original. 5) Enabled PNP in the BIOS (that's still on 'cause I thought I should probably have had it on anyway). 6) Disabled network hardware in the BIOS, put in Linksys PCI NIC. Nope, so I pulled the Linksys and reenabled the MB NIC. 7) Tried a different PS/2 mouse and tried my original mouse on another system. All mice seem to work properly. 8) I tried to get a serial mouse to work, but I couldn't get the serial mouse to work. I can't set the baud rate down to 1200 on this MB. Weird...the lowest it will go is 9600 (verified with minicom). I might play with this again to see if I can isolate the PS/2 mouse hardware on the MB (and the software in Linux). I don't have a USB mouse and Staples didn't have any when I visited. Here is what else I did: 9) Ran memtest for 24 hours - no failures 10) Access the offending system from over the network and play the game - - no lockup! 11) The beer_golf game is a swf file, so I downloaded the file and played it locally - no lockup! 12) Only when I play beer_golf in konqueror, over the net, does the system lockup reliably. 13) CPU temps measured independently from the MB are in the low 40 degree C range. Now, with regard to the serial console, I did get that working at least to a certain point. I've connected the serial cable to another Linux system running minicom. I can see the boot and shutdown messages in two locations (minicom/remote and local). However, when the system is running, I don't see the kernel messages on the serial console. When booting from grub, I have "console=tty0 console=ttyS1,115200n8" in that order. I do see the kernel messages in /var/log/messages and in the xconsole window. - From minicom, if I sent the Break character when the system is running OK (before the lockup occurs), that acts like Alt-SysReq. So, Break-? gives me the SysRq help line. Break-T (show tasks) puts output only on the local system (can't see it on minicom). I am not running getty on this ttyS1 since it seems to interfere with my attempts to use the MagicSysReq key. When it was enabled, I did get successfully logged in. When I put the system at run level 1, then all of the Alt-SysReq output goes to the serial console (minicom can see them!). Now, here's the part that I think might be bad. When I run the program which makes the system lock up, the lockup occurs. I then try to use minicom, and I get no output whatsoever. I can't even get the SysRq help output. I fear this may mean I have a real hardware lockup and not a Linux kernel problem. I know this is a lot to digest, and I really do appreciate all of the feedback. Does anybody have any comments that might help to further isolate this problem? Is there anything that I should try which I have not tried? Thanks!! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCtPa7Hl0iXDssISsRAuZaAJ4mTMd8eSJEPo5LCCiqA8/rqazAQQCfSYmb mHIxlOPglV7Cq42hNKejVxU= =GJzk -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 HI everybody, Perhaps I have found some light in this tunnel. I upgraded to the 2.6.12 kernel (yeah, I know, that just changed the entire experiment) and turned off "CPU freq scaling" and "PCI hotplug support" since I didn't think I needed either of those. I built the 2.6.12 kernel and played beer_golf 5 times in a row and the system DID NOT lockup. I'm not sure why those features were enabled in the first place. (I should probably go back to 2.6.11.11 and disable these two features and see if that has any positive effect.) I'm far too much of a pessimist to claim victory, but perhaps my luck is turning for the better. My fingers are still crossed. I am (extremely) cautiously optimistic. I thought I'd mention this in case somebody reads this thread in the future and maybe, just maybe, this tidbit helps. Besides, I've read a lot of threads where the problem is described in detail, but the answer is never forthcoming(!). Later, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCtkacHl0iXDssISsRAvlGAJ9eVRN8NVKF7sWQtvIfdp4sqLi7BgCfXquw ZP3/O9spJsGk+B2aty2RMDg= =BGSi -----END PGP SIGNATURE-----
In reguards to seeing messages on minicom, what logger do you have installed? Can you just shovel kernel messages over to ttyS01? eg sudo tailf /var/log/kernel ttyS01 On 6/20/05, Andy Stewart <andystewart@comcast.net> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
HI everybody,
Perhaps I have found some light in this tunnel.
I upgraded to the 2.6.12 kernel (yeah, I know, that just changed the entire experiment) and turned off "CPU freq scaling" and "PCI hotplug support" since I didn't think I needed either of those. I built the 2.6.12 kernel and played beer_golf 5 times in a row and the system DID NOT lockup. I'm not sure why those features were enabled in the first place. (I should probably go back to 2.6.11.11 <http://2.6.11.11> and disable these two features and see if that has any positive effect.)
I'm far too much of a pessimist to claim victory, but perhaps my luck is turning for the better. My fingers are still crossed. I am (extremely) cautiously optimistic.
I thought I'd mention this in case somebody reads this thread in the future and maybe, just maybe, this tidbit helps. Besides, I've read a lot of threads where the problem is described in detail, but the answer is never forthcoming(!).
Later,
Andy
- -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFCtkacHl0iXDssISsRAvlGAJ9eVRN8NVKF7sWQtvIfdp4sqLi7BgCfXquw ZP3/O9spJsGk+B2aty2RMDg= =BGSi -----END PGP SIGNATURE----- _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Eric Martin wrote:
In reguards to seeing messages on minicom, what logger do you have installed? Can you just shovel kernel messages over to ttyS01? eg sudo tailf /var/log/kernel ttyS01
I did that per your suggestion and it works just fine. Thanks! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD4DBQFCuIeiHl0iXDssISsRAvu0AJ9meIbihDK9rwqbJ0NFOpV/t/dNkACYqFTA T+XthIh4VSCYVDrFOjSzfg== =gsn3 -----END PGP SIGNATURE-----
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds: andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 andystewart> Eric Martin wrote:
In reguards to seeing messages on minicom, what logger do you have installed? Can you just shovel kernel messages over to ttyS01? eg sudo tailf /var/log/kernel ttyS01
andystewart> I did that per your suggestion and it works just fine. Umm, it works just fine so long as the system is up and running. You really want the kernel to be printing directly to the console. Otherwise, when the system locks up and the NMI fires, you still won't get your messages over the serial console. Andy, it looked to me like you had things configured properly. Indeed, I use a very similar setup all of the time. The only difference is it works for me. (I hate that phrase). I'm not sure what to suggest at this point. If you do an alt-sysrq-h, does that at least print out over the serial console? -Jeff
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote:
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds:
andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
andystewart> Eric Martin wrote:
In reguards to seeing messages on minicom, what logger do you have installed? Can you just shovel kernel messages over to ttyS01? eg sudo tailf /var/log/kernel ttyS01
andystewart> I did that per your suggestion and it works just fine.
Umm, it works just fine so long as the system is up and running. You really want the kernel to be printing directly to the console. Otherwise, when the system locks up and the NMI fires, you still won't get your messages over the serial console.
Andy, it looked to me like you had things configured properly. Indeed, I use a very similar setup all of the time. The only difference is it works for me. (I hate that phrase).
I'm not sure what to suggest at this point. If you do an alt-sysrq-h, does that at least print out over the serial console?
-Jeff
HI Jeff, If I do an alt-sysrq-h, I get the one line help on the serial console. If I do alt-sysrq-m to show memory, I see "SysRq : Show Memory" ont the serial console, but nothing more (unless I use Eric's trick). If I go to runlevel 1, alt-sysrq-m has the entire output on the serial console. Later, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCuI64Hl0iXDssISsRAgYoAJwO2tnZkw6ox+URA61Lh16lyUhE4wCeOedC wQJZTh9CV8pUFPfEHkDYjw8= =H2Yg -----END PGP SIGNATURE-----
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds: andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 andystewart> Jeff Moyer wrote:
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds:
andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
andystewart> Eric Martin wrote:
In reguards to seeing messages on minicom, what logger do you have installed? Can you just shovel kernel messages over to ttyS01? eg sudo tailf /var/log/kernel ttyS01
andystewart> I did that per your suggestion and it works just fine.
Umm, it works just fine so long as the system is up and running. You really want the kernel to be printing directly to the console. Otherwise, when the system locks up and the NMI fires, you still won't get your messages over the serial console.
Andy, it looked to me like you had things configured properly. Indeed, I use a very similar setup all of the time. The only difference is it works for me. (I hate that phrase).
I'm not sure what to suggest at this point. If you do an alt-sysrq-h, does that at least print out over the serial console?
-Jeff
andystewart> HI Jeff, andystewart> If I do an alt-sysrq-h, I get the one line help on the serial andystewart> console. If I do alt-sysrq-m to show memory, I see "SysRq : andystewart> Show Memory" ont the serial console, but nothing more (unless andystewart> I use Eric's trick). If I go to runlevel 1, alt-sysrq-m has andystewart> the entire output on the serial console. You have all of the luck. How about trying netconsole? It's easy enough to setup. On the host which will receive the logs, do something like this: nc -u -l -p 6666 On the problematic system, load the netconsole module: modprobe netconsole netconsole=@<ip-of-this-machine>/<ethX>,6666@<ip-of-server>/<MAC-address-of-server> You can try this or not, depending on whether you still experience crashes. It sounds like the latest kernel fixed things right up for you, right? Oh well, good to have in your bag of tricks, I guess. -Jeff
Right, I forgot about that jeff. If you're running syslog-ng, you can go into /etc/syslog-ng/syslog-ng.conf and add a line: destination console_all { ("/dev/ttyS01"); }; although, I'm not sure if you're locking up if syslog-ng will keep scrolling. I don't know if netcat (nc) will either... On 6/21/05, Jeff Moyer <jmoyer@redhat.com> wrote:
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart < andystewart@comcast.net> adds:
andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
andystewart> Jeff Moyer wrote:
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart@comcast.net> adds:
andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
andystewart> Eric Martin wrote:
In reguards to seeing messages on minicom, what logger do you have installed? Can you just shovel kernel messages over to ttyS01? eg sudo tailf /var/log/kernel ttyS01
andystewart> I did that per your suggestion and it works just fine.
Umm, it works just fine so long as the system is up and running. You really want the kernel to be printing directly to the console. Otherwise, when the system locks up and the NMI fires, you still won't get your messages over the serial console.
Andy, it looked to me like you had things configured properly. Indeed, I use a very similar setup all of the time. The only difference is it works for me. (I hate that phrase).
I'm not sure what to suggest at this point. If you do an alt-sysrq-h, does that at least print out over the serial console?
-Jeff
andystewart> HI Jeff,
andystewart> If I do an alt-sysrq-h, I get the one line help on the serial andystewart> console. If I do alt-sysrq-m to show memory, I see "SysRq : andystewart> Show Memory" ont the serial console, but nothing more (unless andystewart> I use Eric's trick). If I go to runlevel 1, alt-sysrq-m has andystewart> the entire output on the serial console.
You have all of the luck. How about trying netconsole? It's easy enough to setup. On the host which will receive the logs, do something like this:
nc -u -l -p 6666
On the problematic system, load the netconsole module:
modprobe netconsole netconsole=@<ip-of-this-machine>/<ethX>,6666@<ip-of-server>/<MAC-address-of-server>
You can try this or not, depending on whether you still experience crashes. It sounds like the latest kernel fixed things right up for you, right?
Oh well, good to have in your bag of tricks, I guess.
-Jeff _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Eric Martin <freak4uxxx@gmail.com> adds: freak4uxxx> Right, I forgot about that jeff. If you're running syslog-ng, freak4uxxx> you can go into /etc/syslog-ng/syslog-ng.conf and add a line: freak4uxxx> destination console_all { ("/dev/ttyS01"); }; freak4uxxx> although, I'm not sure if you're locking up if syslog-ng will freak4uxxx> keep scrolling. I don't know if netcat (nc) will either... syslog-ng is user space. The kernel should do all of its console writes itself. Your suggestion does not help in the hung system case. Netcat, used as I described (on the system that is receiving messages), is fine. The kernel is outputting the log messages over the network via the netpoll infrastructure. That infrastructure can function even when the system is not servicing interrupts. -Jeff freak4uxxx> On 6/21/05, Jeff Moyer <jmoyer@redhat.com> wrote:
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart < andystewart@comcast.net> adds:
andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
andystewart> Jeff Moyer wrote:
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart >> <andystewart@comcast.net> adds:
andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
andystewart> Eric Martin wrote:
In reguards to seeing messages on minicom, what logger do you have installed? Can you just shovel kernel messages over to ttyS01? eg sudo tailf /var/log/kernel ttyS01
Umm, it works just fine so long as the system is up and running. You really want the kernel to be printing directly to the console. >> Otherwise, when the system locks up and the NMI fires, you still won't get your messages over the serial console.
Andy, it looked to me like you had things configured
andystewart> I did that per your suggestion and it works just fine. properly. Indeed, >> I use a very similar setup all of the time. The only difference is it >> works for me. (I hate that phrase).
I'm not sure what to suggest at this point. If you do an alt-sysrq-h, does that at least print out over the serial console?
-Jeff
andystewart> HI Jeff,
andystewart> If I do an alt-sysrq-h, I get the one line help on the serial andystewart> console. If I do alt-sysrq-m to show memory, I see "SysRq : andystewart> Show Memory" ont the serial console, but nothing more (unless andystewart> I use Eric's trick). If I go to runlevel 1, alt-sysrq-m has andystewart> the entire output on the serial console.
You have all of the luck. How about trying netconsole? It's easy enough to setup. On the host which will receive the logs, do something like this:
nc -u -l -p 6666
On the problematic system, load the netconsole module:
modprobe netconsole netconsole=@<ip-of-this-machine>/<ethX>,6666@<ip-of-server>/<MAC-address-of-server>
You can try this or not, depending on whether you still experience crashes. It sounds like the latest kernel fixed things right up for you, right?
Oh well, good to have in your bag of tricks, I guess.
-Jeff _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
freak4uxxx> Right, I forgot about that jeff. If you're running freak4uxxx> syslog-ng, you can go into /etc/syslog-ng/syslog-ng.conf and freak4uxxx> add a line:<br> <br> <div style="margin-left: 40px;"><span freak4uxxx> style="font-style: italic;">destination console_all { freak4uxxx> ("/dev/ttyS01"); };<br> </span></div> <br> although, freak4uxxx> I'm not sure if you're locking up if syslog-ng will keep freak4uxxx> scrolling. I don't know if netcat (nc) will freak4uxxx> either...<span style="font-style: italic;"></span><br> <span freak4uxxx> style="font-style: italic;"></span><br><div><span freak4uxxx> class="gmail_quote">On 6/21/05, <b freak4uxxx> class="gmail_sendername">Jeff Moyer</b> <<a freak4uxxx> href="mailto:jmoyer@redhat.com">jmoyer@redhat.com</a>> freak4uxxx> wrote:</span><blockquote class="gmail_quote" freak4uxxx> style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt freak4uxxx> 0pt 0pt 0.8ex; padding-left: 1ex;"> ==> Regarding Re: [Wlug] freak4uxxx> HOWTO debug hard lockups; Andy Stewart <<a freak4uxxx> href="mailto:andystewart@comcast.net">andystewart@comcast.net</a>> freak4uxxx> adds:<br><br>andystewart> -----BEGIN PGP SIGNED MESSAGE----- freak4uxxx> Hash: SHA1<br> <br>andystewart> Jeff Moyer freak4uxxx> wrote:<br>>> ==> Regarding Re: [Wlug] HOWTO debug hard freak4uxxx> lockups; Andy Stewart<br>>> <<a freak4uxxx> href="mailto:andystewart@comcast.net">andystewart@comcast.net</a>> freak4uxxx> adds:<br>>> <br>andystewart> -----BEGIN PGP SIGNED freak4uxxx> MESSAGE----- Hash: SHA1<br>>><br>andystewart> Eric freak4uxxx> Martin wrote:<br>>>>> In reguards to seeing freak4uxxx> messages on minicom, what logger do you freak4uxxx> have<br>>>>> installed? Can you just freak4uxxx> shovel kernel messages over to ttyS01? eg freak4uxxx> <br>>>>> sudo tailf /var/log/kernel freak4uxxx> ttyS01<br>>><br>andystewart> I did that per your freak4uxxx> suggestion and it works just fine.<br>>> Umm, it works freak4uxxx> just fine so long as the system is up and freak4uxxx> running. You <br>>> really want the kernel to freak4uxxx> be printing directly to the console.<br>>> Otherwise, freak4uxxx> when the system locks up and the NMI fires, you still freak4uxxx> won't<br>>> get your messages over the serial freak4uxxx> console.<br>>> <br>>> Andy, it looked to me like freak4uxxx> you had things configured freak4uxxx> properly. Indeed,<br>>> I use a very similar freak4uxxx> setup all of the time. The only difference is freak4uxxx> it<br>>> works for me. (I hate that freak4uxxx> phrase).<br>>> <br>>> I'm not sure what to suggest freak4uxxx> at this point. If you do an alt-sysrq-h,<br>>> freak4uxxx> does that at least print out over the serial freak4uxxx> console?<br>>><br>>> freak4uxxx> -Jeff<br>>><br><br>andystewart> HI Jeff, freak4uxxx> <br><br>andystewart> If I do an alt-sysrq-h, I get the one freak4uxxx> line help on the serial<br>andystewart> freak4uxxx> console. If I do alt-sysrq-m to show memory, I see freak4uxxx> "SysRq :<br>andystewart> Show Memory" ont the freak4uxxx> serial console, but nothing more (unless <br>andystewart> I freak4uxxx> use Eric's trick). If I go to runlevel 1, freak4uxxx> alt-sysrq-m has<br>andystewart> the entire output on the freak4uxxx> serial console.<br><br>You have all of the luck. How freak4uxxx> about trying netconsole? It's easy enough <br>to freak4uxxx> setup. On the host which will receive the logs, do freak4uxxx> something like this:<br><br>nc -u -l -p 6666<br><br>On the freak4uxxx> problematic system, load the netconsole module:<br><br>modprobe freak4uxxx> netconsole freak4uxxx> netconsole=@<ip-of-this-machine>/<ethX>,6666@<ip-of-server>/<MAC-address-of-server> freak4uxxx> <br><br><br>You can try this or not, depending on whether you freak4uxxx> still experience<br>crashes. It sounds like the freak4uxxx> latest kernel fixed things right up for freak4uxxx> you,<br>right?<br><br>Oh well, good to have in your bag of freak4uxxx> tricks, I guess. freak4uxxx> <br><br>-Jeff<br>_______________________________________________<br>Wlug freak4uxxx> mailing list<br><a freak4uxxx> href="mailto:Wlug@mail.wlug.org">Wlug@mail.wlug.org</a><br><a freak4uxxx> href="http://mail.wlug.org/mailman/listinfo/wlug">http://mail.wlug.org/mailman/listinfo/wlug freak4uxxx> </a><br></blockquote></div><br>
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andy Stewart wrote:
HI Jeff,
If I do an alt-sysrq-h, I get the one line help on the serial console. If I do alt-sysrq-m to show memory, I see "SysRq : Show Memory" ont the serial console, but nothing more (unless I use Eric's trick). If I go to runlevel 1, alt-sysrq-m has the entire output on the serial console.
Later,
Andy
HI Everybody, The problem of the serial console not responding as I expected has been solved. It turns out that on my system, in /etc/sysconfig/syslog, the loglevel is set to 1. If it is set to 8, then all of the kernel messages will come out on the serial console. I set this with the MagicSysReq key and now it works as I expect. Many thanks to Jeff Moyer for his help with this problem. Later, Andy
-- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org
_______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCugdJHl0iXDssISsRAq7FAJ9c93PzhdT2YAahG50wHRTKCkTOtgCeOCox ahEmSJk/eDlUH7JwXHZDiBE= =Q8ta -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andy Stewart wrote:
HI everybody,
Perhaps I have found some light in this tunnel.
I upgraded to the 2.6.12 kernel (yeah, I know, that just changed the entire experiment) and turned off "CPU freq scaling" and "PCI hotplug support" since I didn't think I needed either of those. I built the 2.6.12 kernel and played beer_golf 5 times in a row and the system DID NOT lockup. I'm not sure why those features were enabled in the first place. (I should probably go back to 2.6.11.11 and disable these two features and see if that has any positive effect.)
I'm far too much of a pessimist to claim victory, but perhaps my luck is turning for the better. My fingers are still crossed. I am (extremely) cautiously optimistic.
I thought I'd mention this in case somebody reads this thread in the future and maybe, just maybe, this tidbit helps. Besides, I've read a lot of threads where the problem is described in detail, but the answer is never forthcoming(!).
Later,
Andy
Hi Everybody, I went back to the 2.6.11.11 kernel and turned off the "CPU freq scaling" and "PCI hotplug support", but it didn't matter, as this kernel still locks up on me very quickly with the beer_golf.swf game (a copy of which I've saved as a diagnostic for future kernels to which I might upgrade - a *long* time from now). I am therefore concluding that something in 2.6.12 has fixed this problem since I have not yet locked up this kernel (under any conditions). I did notice several K8 and AMD64 specific notes in the CHANGELOG for this kernel. Thanks again(!!) for all of your helpful suggestions. Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCwMEEHl0iXDssISsRAsRgAJ0ZDt/dlQrN+t1KRvgoBoC97yhcmgCfT7Vo 31dPPiB3sSKrDsx/FjhHYzs= =7kuI -----END PGP SIGNATURE-----
participants (11)
-
Andy Stewart -
Baris Hasdemir -
Chuck R. Anderson -
Eric Martin -
Gary Hanley -
Gary＠home -
J Schonberg -
Jeff Moyer -
John Stoffel -
Michael Long -
Walt Sawyer