WLUG
By thread
wlug@lists.wlug.org
By month
Messages by month
- ----- 2026 -----
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2008 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2007 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2006 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2005 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2004 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2003 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2002 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2001 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2000 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
June 2005
- 24 participants
- 64 messages
RE: [Wlug] HOWTO debug hard lockups
by Keller, Tim
Andy,
Your lockup's sound like an SMP issue. I'd try booting a non SMP kernel
and then seeing if you can duplicate the lockup...
I say this because I was getting practically the same behavior with a
duel Xeon machine running FC3. The machine would just up and die in the
same way your machine is.
Just my $.02.
Tim.
-----Original Message-----
From: wlug-bounces(a)mail.wlug.org [mailto:wlug-bounces@mail.wlug.org] On
Behalf Of Andy Stewart
Sent: Wednesday, June 08, 2005 7:15 PM
To: jmoyer(a)redhat.com; Worcester Linux Users Group
Subject: Re: [Wlug] HOWTO debug hard lockups
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Jeff Moyer wrote:
> ==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart
<andystewart(a)comcast.net> adds:
>
> andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>
> andystewart> Jeff Moyer wrote:
>>>==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart
>>><andystewart(a)comcast.net> adds:
>>>
> andystewart> HI gang,
> andystewart> My dual Opteron machine is not happy. I cannot get more
than
> andystewart> 7 straight days of uptime without getting a hard lock,
> andystewart> requiring a reboot. (My definition of hard lock is:
machine
> andystewart> responds neither to keyboard input, mouse input, nor
network
> andystewart> pings).
> andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I
had
> andystewart> 3 tonight, and 3-4 on a previous occasion while running
> andystewart> OpenOffice). It makes no sense to me that an application
run
> andystewart> as a normal user could lockup a machine.
> andystewart> I've tried setting "nmi_watchdog=1" to see if I could get
an
> andystewart> "oops" when it hard locks - no dice. Do you know any
other
> andystewart> tricks I could try to see if it is the kernel which is
locking
> andystewart> up? I'm running SuSE's version of 2.6.8.
>>>Did you verify that NMIs are being delivered? After boot, cat
>>>/proc/interrupts and make sure the NMI line is non-zero. Also note
>>>that, at least with upstream and Red Hat kernels, the nmi_watchdog
>>>defaults to 1 for Opterons (i.e. you shouldn't need to manually set
it).
>
> andystewart> HI Jeff,
>
> andystewart> Well, this is weird, I'm seeing a ZERO count for NMIs, so
that
> andystewart> makes me think they are NOT being delivered. How would I
go
> andystewart> about solving *that* little problem? I did a "cat
>
> Well, you can try booting with nmi_watchdog=2. This will try to use
the
> local APIC to deliver nmi's, but I haven't actually seen a dual
processor
> system that required this (all of them I've seen work with
nmi_watchdog=1).
> It is worth a try, however.
With nmi_watchdog=1, I see the following output from "dmesg | grep -i
nmi" with the 2.6.11.11 kernel I very recently installed:
Bootdata ok (command line is root=/dev/sda3 vga=0x31a selinux=0
console=tty0 resume=/dev/sda2 desktop elevator=as nmi_watchdog=1
splash=verbose)
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
Kernel command line: root=/dev/sda3 vga=0x31a selinux=0 console=tty0
resume=/dev/sda2 desktop elevator=as nmi_watchdog=1 splash=verbose
activating NMI Watchdog ... done.
testing NMI watchdog ... CPU#1: NMI appears to be stuck (0)!
I'll try nmi_watchdog=2 shortly.
Thanks,
Andy
- --
Andy Stewart, Founder
Worcester Linux Users' Group
Worcester, MA, USA
http://www.wlug.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFCp3v8Hl0iXDssISsRAoDJAJ0TonNTSEAd+4W0GRPwTW/UoGFssACdEEq9
8IgEFJ7VAL/FVeyPXrEapHg=
=jcjE
-----END PGP SIGNATURE-----
_______________________________________________
Wlug mailing list
Wlug(a)mail.wlug.org
http://mail.wlug.org/mailman/listinfo/wlug
June 9, 2005
Re: [Wlug] HOWTO debug hard lockups
by Andy Stewart
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Jeff Moyer wrote:
> ==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds:
>
> andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>
> andystewart> Jeff Moyer wrote:
>>>==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart
>>><andystewart(a)comcast.net> adds:
>>>
> andystewart> HI gang,
> andystewart> My dual Opteron machine is not happy. I cannot get more than
> andystewart> 7 straight days of uptime without getting a hard lock,
> andystewart> requiring a reboot. (My definition of hard lock is: machine
> andystewart> responds neither to keyboard input, mouse input, nor network
> andystewart> pings).
> andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had
> andystewart> 3 tonight, and 3-4 on a previous occasion while running
> andystewart> OpenOffice). It makes no sense to me that an application run
> andystewart> as a normal user could lockup a machine.
> andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an
> andystewart> "oops" when it hard locks - no dice. Do you know any other
> andystewart> tricks I could try to see if it is the kernel which is locking
> andystewart> up? I'm running SuSE's version of 2.6.8.
>>>Did you verify that NMIs are being delivered? After boot, cat
>>>/proc/interrupts and make sure the NMI line is non-zero. Also note
>>>that, at least with upstream and Red Hat kernels, the nmi_watchdog
>>>defaults to 1 for Opterons (i.e. you shouldn't need to manually set it).
>
> andystewart> HI Jeff,
>
> andystewart> Well, this is weird, I'm seeing a ZERO count for NMIs, so that
> andystewart> makes me think they are NOT being delivered. How would I go
> andystewart> about solving *that* little problem? I did a "cat
>
> Well, you can try booting with nmi_watchdog=2. This will try to use the
> local APIC to deliver nmi's, but I haven't actually seen a dual processor
> system that required this (all of them I've seen work with nmi_watchdog=1).
> It is worth a try, however.
With nmi_watchdog=1, I see the following output from "dmesg | grep -i
nmi" with the 2.6.11.11 kernel I very recently installed:
Bootdata ok (command line is root=/dev/sda3 vga=0x31a selinux=0
console=tty0 resume=/dev/sda2 desktop elevator=as nmi_watchdog=1
splash=verbose)
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
Kernel command line: root=/dev/sda3 vga=0x31a selinux=0 console=tty0
resume=/dev/sda2 desktop elevator=as nmi_watchdog=1 splash=verbose
activating NMI Watchdog ... done.
testing NMI watchdog ... CPU#1: NMI appears to be stuck (0)!
I'll try nmi_watchdog=2 shortly.
Thanks,
Andy
- --
Andy Stewart, Founder
Worcester Linux Users' Group
Worcester, MA, USA
http://www.wlug.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFCp3v8Hl0iXDssISsRAoDJAJ0TonNTSEAd+4W0GRPwTW/UoGFssACdEEq9
8IgEFJ7VAL/FVeyPXrEapHg=
=jcjE
-----END PGP SIGNATURE-----
June 8, 2005
Re: [Wlug] HOWTO debug hard lockups
by Michael Long
Andy Stewart wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>John Stoffel wrote:
>
>
>>Andy,
>>
>>The best thing to do here is to run a serial console to another one of
>>your systems and use THAT to try and catch the issue. You can then do
>>a SysRq from the remote serial console and/or catch any error
>>messages.
>>
>>Also, are there any messages in dmesg or /var/log/messages? How does
>>the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see
>>if the problem repeats.
>>
>>John
>>
>>
>
>HI John,
>
>I'm going to give the serial console a try - I may have a cable lying
>around.
>
>I upgraded to the 2.6.11.11 kernel which did have several fixes for K8.
> We'll see how that goes.
>
>Thanks for the ideas!
>
>Andy
>
>- --
>Andy Stewart, Founder
>Worcester Linux Users' Group
>Worcester, MA, USA
>http://www.wlug.org
>
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v1.2.5 (GNU/Linux)
>Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>
>iD8DBQFCp2yhHl0iXDssISsRAqxcAJwLntOh9GpNmefW/+JdwbAUMRJKBACfbV1P
>s2eqMUnY6rjZaIfI6MzZLqQ=
>=aLfl
>-----END PGP SIGNATURE-----
>_______________________________________________
>Wlug mailing list
>Wlug(a)mail.wlug.org
>http://mail.wlug.org/mailman/listinfo/wlug
>
>
Andy,
If that doesn't help try using the system without the kvm and see if you
get the same behavior. I have seen odd things in the past when using a
faulty kvm switch.
Mike
June 8, 2005
Re: [Wlug] HOWTO debug hard lockups
by Andy Stewart
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
John Stoffel wrote:
> Andy,
>
> The best thing to do here is to run a serial console to another one of
> your systems and use THAT to try and catch the issue. You can then do
> a SysRq from the remote serial console and/or catch any error
> messages.
>
> Also, are there any messages in dmesg or /var/log/messages? How does
> the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see
> if the problem repeats.
>
> John
HI John,
I'm going to give the serial console a try - I may have a cable lying
around.
I upgraded to the 2.6.11.11 kernel which did have several fixes for K8.
We'll see how that goes.
Thanks for the ideas!
Andy
- --
Andy Stewart, Founder
Worcester Linux Users' Group
Worcester, MA, USA
http://www.wlug.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFCp2yhHl0iXDssISsRAqxcAJwLntOh9GpNmefW/+JdwbAUMRJKBACfbV1P
s2eqMUnY6rjZaIfI6MzZLqQ=
=aLfl
-----END PGP SIGNATURE-----
June 8, 2005
Job opportunity in Natick for experienced Unix admin
by Simoncini, Matthew
List,
I work for a large consulting company in Natick, MA. and we're looking for a
couple of experienced (mid level) Unix administrators to join our team. Some
of the skills that are required include:
Unix experience (Sun Solaris 8 or higher)
Veritas volume manager
Veritas cluster services
Storage management experience (preferably EMC Clarion or DMX)
Backup management experience (preferably EMC EDM)
Team player
Desire to learn and grow
If you or anyone you know is looking for a position as a Unix Administrator,
please contact me off list at matthewsmbox-resumes(a)yahoo.com
<mailto:matthewsmbox-resumes@yahoo.com> .
Thanks.
Matthew
June 8, 2005
MySQL curses management tool?
by John Stoffel
Hi all,
A quick googling didn't help, so I'm turning to you all. I'm looking
for a mysql management tool which is CLI/curses based and more
powerful/easier to use when compared to the mysqlaccess and mysqladmin
stuff.
Basically, something to help me manage DBs, users and permissions over
an SSH connection. I know I could use some of the X tools, but I'd
prefer not too if possible.
Thanks,
John
June 8, 2005
Re: [Wlug] HOWTO debug hard lockups
by John Stoffel
Andy,
The best thing to do here is to run a serial console to another one of
your systems and use THAT to try and catch the issue. You can then do
a SysRq from the remote serial console and/or catch any error
messages.
Also, are there any messages in dmesg or /var/log/messages? How does
the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see
if the problem repeats.
John
June 8, 2005
Re: [Wlug] HOWTO debug hard lockups
by Jeff Moyer
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds:
andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
andystewart> Jeff Moyer wrote:
>> ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart
>> <andystewart(a)comcast.net> adds:
>>
andystewart> HI gang,
>>
andystewart> My dual Opteron machine is not happy. I cannot get more than
andystewart> 7 straight days of uptime without getting a hard lock,
andystewart> requiring a reboot. (My definition of hard lock is: machine
andystewart> responds neither to keyboard input, mouse input, nor network
andystewart> pings).
>>
andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had
andystewart> 3 tonight, and 3-4 on a previous occasion while running
andystewart> OpenOffice). It makes no sense to me that an application run
andystewart> as a normal user could lockup a machine.
>>
andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an
andystewart> "oops" when it hard locks - no dice. Do you know any other
andystewart> tricks I could try to see if it is the kernel which is locking
andystewart> up? I'm running SuSE's version of 2.6.8.
>> Did you verify that NMIs are being delivered? After boot, cat
>> /proc/interrupts and make sure the NMI line is non-zero. Also note
>> that, at least with upstream and Red Hat kernels, the nmi_watchdog
>> defaults to 1 for Opterons (i.e. you shouldn't need to manually set it).
andystewart> HI Jeff,
andystewart> Well, this is weird, I'm seeing a ZERO count for NMIs, so that
andystewart> makes me think they are NOT being delivered. How would I go
andystewart> about solving *that* little problem? I did a "cat
Well, you can try booting with nmi_watchdog=2. This will try to use the
local APIC to deliver nmi's, but I haven't actually seen a dual processor
system that required this (all of them I've seen work with nmi_watchdog=1).
It is worth a try, however.
andystewart> /proc/cmdline" to insure that I had "nmi_watchdog=1" and
andystewart> indeed it is there. Perhaps this is the clue we've been
andystewart> seeking.
Well, it only tells you that the nmi_watchdog won't trigger. We still have
no insight into what the problem might actually be.
>> If the NMI watchdog works, it will print a message to the console.
>> However, you will not see this if you are in X windows. Do you have a
>> serial console hooked up, by any chance? I strongly suggest it if you
>> have the means.
andystewart> I think I have a cable to which I could connect the serial
andystewart> port of the Opteron to the serial port of another Linux box
andystewart> (and then use minicom or some such terminal program).
>> Aside from this, if it is indeed a hard lockup, there is really nothing
>> you can do (without purchasing other hardware to help debug the
>> problem).
andystewart> Yeah, I was afraid of that.
>> Please give these suggestions a shot and let us know how it goes.
andystewart> Shall do - thanks, everybody!
andystewart> Oh, be advised that when you smash your fist on the keyboard
andystewart> after your system locks up for the umpteenth time, that a lot
andystewart> of dead skin cells will come flying upward out of the bowels
andystewart> of the keyboard. I would recommend safety glasses.
Noted. =)
-Jeff
June 8, 2005
Re: [Wlug] HOWTO debug hard lockups
by Jeff Moyer
==> Regarding Re: [Wlug] HOWTO debug hard lockups; Walt Sawyer <wsawyer(a)norfolk-county.com> adds:
wsawyer> Andy, Are you sure it's the hard drive? Try running Knoppix live
wsawyer> CD for a while. If it's still running after days(max uptime)+5,
wsawyer> then perhaps it's something else.
This changes things entirely. He'll be running a different kernel, so this
test is not useful, in my opinion.
-Jeff
wsawyer> If this message is redundant, ignore it! Walt
wsawyer> On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>
>> Jeff Moyer wrote: > ==> Regarding [Wlug] HOWTO debug hard lockups; Andy
>> Stewart <andystewart(a)comcast.net> adds:
>> >
>> > andystewart> HI gang,
>> >
>> > andystewart> My dual Opteron machine is not happy. I cannot get more
>> than > andystewart> 7 straight days of uptime without getting a hard
>> lock, > andystewart> requiring a reboot. (My definition of hard lock
>> is: machine > andystewart> responds neither to keyboard input, mouse
>> input, nor network > andystewart> pings).
>> >
>> > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I
>> had > andystewart> 3 tonight, and 3-4 on a previous occasion while
>> running > andystewart> OpenOffice). It makes no sense to me that an
>> application run > andystewart> as a normal user could lockup a machine.
>> >
>> > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get
>> an > andystewart> "oops" when it hard locks - no dice. Do you know any
>> other > andystewart> tricks I could try to see if it is the kernel which
>> is locking > andystewart> up? I'm running SuSE's version of 2.6.8.
>> >
>> > Did you verify that NMIs are being delivered? After boot, cat >
>> /proc/interrupts and make sure the NMI line is non-zero. Also note
>> that, > at least with upstream and Red Hat kernels, the nmi_watchdog
>> defaults to 1 > for Opterons (i.e. you shouldn't need to manually set
>> it).
>>
>> HI Jeff,
>>
>> Well, this is weird, I'm seeing a ZERO count for NMIs, so that makes me
>> think they are NOT being delivered. How would I go about solving *that*
>> little problem? I did a "cat /proc/cmdline" to insure that I had
>> "nmi_watchdog=1" and indeed it is there. Perhaps this is the clue we've
>> been seeking.
>>
>> > If the NMI watchdog works, it will print a message to the console. >
>> However, you will not see this if you are in X windows. Do you have a >
>> serial console hooked up, by any chance? I strongly suggest it if you
>> have > the means.
>>
>> I think I have a cable to which I could connect the serial port of the
>> Opteron to the serial port of another Linux box (and then use minicom or
>> some such terminal program).
>>
>> >
>> > Aside from this, if it is indeed a hard lockup, there is really
>> nothing you > can do (without purchasing other hardware to help debug
>> the problem).
>>
>> Yeah, I was afraid of that.
>>
>> >
>> > Please give these suggestions a shot and let us know how it goes.
>>
>> Shall do - thanks, everybody!
>>
>> Oh, be advised that when you smash your fist on the keyboard after your
>> system locks up for the umpteenth time, that a lot of dead skin cells
>> will come flying upward out of the bowels of the keyboard. I would
>> recommend safety glasses.
>>
>> Later,
>>
>> Andy
>>
>> - --
>> Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA
>> http://www.wlug.org
>>
>> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment:
>> Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>>
>> iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV
>> u6O8n+PSW4M0I1MHXsH06Xo= =ViVg -----END PGP SIGNATURE-----
>> _______________________________________________ Wlug mailing list
>> Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
>>
wsawyer> -- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
wsawyer> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html;
wsawyer> CHARSET=UTF-8"> <META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2">
wsawyer> </HEAD> <BODY> Andy,<BR> Are you sure it's the hard drive?
wsawyer> Try running Knoppix live CD for a while. If it's still
wsawyer> running after days(max uptime)+5, then perhaps it's something
wsawyer> else.<BR> <BR> If this message is redundant, ignore it!<BR>
wsawyer> Walt<BR> <BR> On Tue, 2005-06-07 at 19:19 -0400, Andy Stewart
wsawyer> wrote: <BLOCKQUOTE TYPE=CITE> <PRE> <FONT
wsawyer> COLOR="#000000">-----BEGIN PGP SIGNED MESSAGE-----</FONT> <FONT
wsawyer> COLOR="#000000">Hash: SHA1</FONT>
wsawyer> <FONT COLOR="#000000">Jeff Moyer wrote:</FONT> <FONT
wsawyer> COLOR="#000000">> ==> Regarding [Wlug] HOWTO debug hard
wsawyer> lockups; Andy Stewart <<A
wsawyer> HREF="mailto:andystewart@comcast.net">andystewart(a)comcast.net</A>>
wsawyer> adds:</FONT> <FONT COLOR="#000000">> </FONT> <FONT
wsawyer> COLOR="#000000">> andystewart> HI gang,</FONT> <FONT
wsawyer> COLOR="#000000">> </FONT> <FONT COLOR="#000000">>
wsawyer> andystewart> My dual Opteron machine is not happy. I cannot
wsawyer> get more than</FONT> <FONT COLOR="#000000">> andystewart> 7
wsawyer> straight days of uptime without getting a hard lock,</FONT> <FONT
wsawyer> COLOR="#000000">> andystewart> requiring a reboot. (My
wsawyer> definition of hard lock is: machine</FONT> <FONT
wsawyer> COLOR="#000000">> andystewart> responds neither to keyboard
wsawyer> input, mouse input, nor network</FONT> <FONT COLOR="#000000">>
wsawyer> andystewart> pings).</FONT> <FONT COLOR="#000000">> </FONT>
wsawyer> <FONT COLOR="#000000">> andystewart> I can stimulate hard
wsawyer> locks by running OpenOffice 1.1.3 (I had</FONT> <FONT
wsawyer> COLOR="#000000">> andystewart> 3 tonight, and 3-4 on a
wsawyer> previous occasion while running</FONT> <FONT COLOR="#000000">>
wsawyer> andystewart> OpenOffice). It makes no sense to me that an
wsawyer> application run</FONT> <FONT COLOR="#000000">> andystewart>
wsawyer> as a normal user could lockup a machine.</FONT> <FONT
wsawyer> COLOR="#000000">> </FONT> <FONT COLOR="#000000">>
wsawyer> andystewart> I've tried setting "nmi_watchdog=1" to
wsawyer> see if I could get an</FONT> <FONT COLOR="#000000">>
wsawyer> andystewart> "oops" when it hard locks - no dice. Do
wsawyer> you know any other</FONT> <FONT COLOR="#000000">>
wsawyer> andystewart> tricks I could try to see if it is the kernel
wsawyer> which is locking</FONT> <FONT COLOR="#000000">> andystewart>
wsawyer> up? I'm running SuSE's version of 2.6.8.</FONT> <FONT
wsawyer> COLOR="#000000">> </FONT> <FONT COLOR="#000000">> Did you
wsawyer> verify that NMIs are being delivered? After boot, cat</FONT>
wsawyer> <FONT COLOR="#000000">> /proc/interrupts and make sure the NMI
wsawyer> line is non-zero. Also note that,</FONT> <FONT
wsawyer> COLOR="#000000">> at least with upstream and Red Hat kernels,
wsawyer> the nmi_watchdog defaults to 1</FONT> <FONT COLOR="#000000">>
wsawyer> for Opterons (i.e. you shouldn't need to manually set it).</FONT>
wsawyer> <FONT COLOR="#000000">HI Jeff,</FONT>
wsawyer> <FONT COLOR="#000000">Well, this is weird, I'm seeing a ZERO count
wsawyer> for NMIs, so that makes me</FONT> <FONT COLOR="#000000">think they
wsawyer> are NOT being delivered. How would I go about solving
wsawyer> *that*</FONT> <FONT COLOR="#000000">little problem? I did a
wsawyer> "cat /proc/cmdline" to insure that I had</FONT> <FONT
wsawyer> COLOR="#000000">"nmi_watchdog=1" and indeed it is there.
wsawyer> Perhaps this is the clue we've</FONT> <FONT COLOR="#000000">been
wsawyer> seeking.</FONT>
wsawyer> <FONT COLOR="#000000">> If the NMI watchdog works, it will
wsawyer> print a message to the console.</FONT> <FONT COLOR="#000000">>
wsawyer> However, you will not see this if you are in X windows. Do you
wsawyer> have a</FONT> <FONT COLOR="#000000">> serial console hooked up,
wsawyer> by any chance? I strongly suggest it if you have</FONT> <FONT
wsawyer> COLOR="#000000">> the means.</FONT>
wsawyer> <FONT COLOR="#000000">I think I have a cable to which I could
wsawyer> connect the serial port of the</FONT> <FONT
wsawyer> COLOR="#000000">Opteron to the serial port of another Linux box
wsawyer> (and then use minicom or</FONT> <FONT COLOR="#000000">some such
wsawyer> terminal program).</FONT>
wsawyer> <FONT COLOR="#000000">> </FONT> <FONT COLOR="#000000">>
wsawyer> Aside from this, if it is indeed a hard lockup, there is really
wsawyer> nothing you</FONT> <FONT COLOR="#000000">> can do (without
wsawyer> purchasing other hardware to help debug the problem).</FONT>
wsawyer> <FONT COLOR="#000000">Yeah, I was afraid of that.</FONT>
wsawyer> <FONT COLOR="#000000">> </FONT> <FONT COLOR="#000000">>
wsawyer> Please give these suggestions a shot and let us know how it
wsawyer> goes.</FONT>
wsawyer> <FONT COLOR="#000000">Shall do - thanks, everybody!</FONT>
wsawyer> <FONT COLOR="#000000">Oh, be advised that when you smash your fist
wsawyer> on the keyboard after your</FONT> <FONT COLOR="#000000">system
wsawyer> locks up for the umpteenth time, that a lot of dead skin
wsawyer> cells</FONT> <FONT COLOR="#000000">will come flying upward out of
wsawyer> the bowels of the keyboard. I would</FONT> <FONT
wsawyer> COLOR="#000000">recommend safety glasses.</FONT>
wsawyer> <FONT COLOR="#000000">Later,</FONT>
wsawyer> <FONT COLOR="#000000">Andy</FONT>
wsawyer> <FONT COLOR="#000000">- --</FONT> <FONT COLOR="#000000">Andy
wsawyer> Stewart, Founder</FONT> <FONT COLOR="#000000">Worcester Linux
wsawyer> Users' Group</FONT> <FONT COLOR="#000000">Worcester, MA,
wsawyer> USA</FONT> <FONT COLOR="#000000"><A
wsawyer> HREF="http://www.wlug.org">http://www.wlug.org</A></FONT>
wsawyer> <FONT COLOR="#000000">-----BEGIN PGP SIGNATURE-----</FONT> <FONT
wsawyer> COLOR="#000000">Version: GnuPG v1.2.5 (GNU/Linux)</FONT> <FONT
wsawyer> COLOR="#000000">Comment: Using GnuPG with Thunderbird - <A
wsawyer> HREF="http://enigmail.mozdev.org">http://enigmail.mozdev.org</A></FONT>
wsawyer> <FONT
wsawyer> COLOR="#000000">iD8DBQFCpiuVHl0iXDssISsRAowjAJsHjggG0QsMPQ/H+2YQnzNZPtF9gQCfepvV</FONT>
wsawyer> <FONT COLOR="#000000">u6O8n+PSW4M0I1MHXsH06Xo=</FONT> <FONT
wsawyer> COLOR="#000000">=ViVg</FONT> <FONT COLOR="#000000">-----END PGP
wsawyer> SIGNATURE-----</FONT> <FONT
wsawyer> COLOR="#000000">_______________________________________________</FONT>
wsawyer> <FONT COLOR="#000000">Wlug mailing list</FONT> <FONT
wsawyer> COLOR="#000000"><A
wsawyer> HREF="mailto:Wlug@mail.wlug.org">Wlug(a)mail.wlug.org</A></FONT>
wsawyer> <FONT COLOR="#000000"><A
wsawyer> HREF="http://mail.wlug.org/mailman/listinfo/wlug">http://mail.wlug.org/mailman/listinfo/wlug</A></FONT>
wsawyer> </PRE> </BLOCKQUOTE> <TABLE CELLSPACING="0" CELLPADDING="0"
wsawyer> WIDTH="100%"> <TR> <TD> <PRE> -- </PRE> </TD> </TR> </TABLE>
wsawyer> </BODY> </HTML> _______________________________________________
wsawyer> Wlug mailing list Wlug(a)mail.wlug.org
wsawyer> http://mail.wlug.org/mailman/listinfo/wlug
June 8, 2005
Re: [Wlug] HOWTO debug hard lockups
by Gary Hanley
On Tue, 7 Jun 2005, Andy Stewart wrote:
> I'd rather not give up KDE, but I could try blackbox for a while to see
> what happens.
Well, my guess is that it is a hardware issue
How about the MB settings, like the memory timings/clock ratios? Sometimes
this can cause problems if they are set too aggressively. Usually there
are one-off options in the BIOS menu for stable settings and speed
settings so you don't have to tweak everything.
Anything there you can check? Maybe down-shift your settings a little and
see if the problem goes away.
I haven't had the pleasure of working with a dual system every so I'm
not familiar with any related BIOS settings you may need to modify.
-- Gary
June 8, 2005