- WLUG - wlug.mailman3.com

Re: [Wlug] HOWTO debug hard lockups

by John Stoffel

Andy> So far, so good, but I only installed 2.6.11.11: Andy> 7:49pm up 2 days 20:40, 7 users, load average: 2.04, 2.08, 2.08 Andy> Linux amdtux 2.6.11.11 #1 SMP Tue Jun 7 21:12:53 EDT 2005 x86_64 x86_64 Andy> x86_64 GNU/Linux I assume you did a 'make oldconfig' to keep your settings a close as possible to the previous kernel? I'm wondering if there's some sort of difference in settings that might have made a difference here. In any case, good luck, and great to hear it's working well for you. My old dual cpu PIII Xeon 550mhz is still chugging along well, but I'm starting to think I want a silent NFS/mysql/daemon server with a bunch of disk and a tape drive to sit off in a quiet corner somewhere to do backups, serve files, etc. And be stable and quiet. Then my main box could become a more general play system again. John

June 12, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 So far, so good, but I only installed 2.6.11.11: 7:49pm up 2 days 20:40, 7 users, load average: 2.04, 2.08, 2.08 Linux amdtux 2.6.11.11 #1 SMP Tue Jun 7 21:12:53 EDT 2005 x86_64 x86_64 x86_64 GNU/Linux - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCqiduHl0iXDssISsRApBGAJ4w8HXNq60qb/6BMON3uz4aoywjRwCdHWrC xVYbnucZfFPBUFt3MWZhXNc= =z/Mt -----END PGP SIGNATURE-----

June 10, 2005

RE: [Wlug] HOWTO debug hard lockups

by Keller, Tim

Andy, Your lockup's sound like an SMP issue. I'd try booting a non SMP kernel and then seeing if you can duplicate the lockup... I say this because I was getting practically the same behavior with a duel Xeon machine running FC3. The machine would just up and die in the same way your machine is. Just my $.02. Tim. -----Original Message----- From: wlug-bounces(a)mail.wlug.org [mailto:wlug-bounces@mail.wlug.org] On Behalf Of Andy Stewart Sent: Wednesday, June 08, 2005 7:15 PM To: jmoyer(a)redhat.com; Worcester Linux Users Group Subject: Re: [Wlug] HOWTO debug hard lockups -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote: > ==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: > > andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > > andystewart> Jeff Moyer wrote: >>>==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart >>><andystewart(a)comcast.net> adds: >>> > andystewart> HI gang, > andystewart> My dual Opteron machine is not happy. I cannot get more than > andystewart> 7 straight days of uptime without getting a hard lock, > andystewart> requiring a reboot. (My definition of hard lock is: machine > andystewart> responds neither to keyboard input, mouse input, nor network > andystewart> pings). > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had > andystewart> 3 tonight, and 3-4 on a previous occasion while running > andystewart> OpenOffice). It makes no sense to me that an application run > andystewart> as a normal user could lockup a machine. > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an > andystewart> "oops" when it hard locks - no dice. Do you know any other > andystewart> tricks I could try to see if it is the kernel which is locking > andystewart> up? I'm running SuSE's version of 2.6.8. >>>Did you verify that NMIs are being delivered? After boot, cat >>>/proc/interrupts and make sure the NMI line is non-zero. Also note >>>that, at least with upstream and Red Hat kernels, the nmi_watchdog >>>defaults to 1 for Opterons (i.e. you shouldn't need to manually set it). > > andystewart> HI Jeff, > > andystewart> Well, this is weird, I'm seeing a ZERO count for NMIs, so that > andystewart> makes me think they are NOT being delivered. How would I go > andystewart> about solving *that* little problem? I did a "cat > > Well, you can try booting with nmi_watchdog=2. This will try to use the > local APIC to deliver nmi's, but I haven't actually seen a dual processor > system that required this (all of them I've seen work with nmi_watchdog=1). > It is worth a try, however. With nmi_watchdog=1, I see the following output from "dmesg | grep -i nmi" with the 2.6.11.11 kernel I very recently installed: Bootdata ok (command line is root=/dev/sda3 vga=0x31a selinux=0 console=tty0 resume=/dev/sda2 desktop elevator=as nmi_watchdog=1 splash=verbose) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) Kernel command line: root=/dev/sda3 vga=0x31a selinux=0 console=tty0 resume=/dev/sda2 desktop elevator=as nmi_watchdog=1 splash=verbose activating NMI Watchdog ... done. testing NMI watchdog ... CPU#1: NMI appears to be stuck (0)! I'll try nmi_watchdog=2 shortly. Thanks, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCp3v8Hl0iXDssISsRAoDJAJ0TonNTSEAd+4W0GRPwTW/UoGFssACdEEq9 8IgEFJ7VAL/FVeyPXrEapHg= =jcjE -----END PGP SIGNATURE----- _______________________________________________ Wlug mailing list Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug

June 9, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Moyer wrote: > ==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: > > andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > > andystewart> Jeff Moyer wrote: >>>==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart >>><andystewart(a)comcast.net> adds: >>> > andystewart> HI gang, > andystewart> My dual Opteron machine is not happy. I cannot get more than > andystewart> 7 straight days of uptime without getting a hard lock, > andystewart> requiring a reboot. (My definition of hard lock is: machine > andystewart> responds neither to keyboard input, mouse input, nor network > andystewart> pings). > andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had > andystewart> 3 tonight, and 3-4 on a previous occasion while running > andystewart> OpenOffice). It makes no sense to me that an application run > andystewart> as a normal user could lockup a machine. > andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an > andystewart> "oops" when it hard locks - no dice. Do you know any other > andystewart> tricks I could try to see if it is the kernel which is locking > andystewart> up? I'm running SuSE's version of 2.6.8. >>>Did you verify that NMIs are being delivered? After boot, cat >>>/proc/interrupts and make sure the NMI line is non-zero. Also note >>>that, at least with upstream and Red Hat kernels, the nmi_watchdog >>>defaults to 1 for Opterons (i.e. you shouldn't need to manually set it). > > andystewart> HI Jeff, > > andystewart> Well, this is weird, I'm seeing a ZERO count for NMIs, so that > andystewart> makes me think they are NOT being delivered. How would I go > andystewart> about solving *that* little problem? I did a "cat > > Well, you can try booting with nmi_watchdog=2. This will try to use the > local APIC to deliver nmi's, but I haven't actually seen a dual processor > system that required this (all of them I've seen work with nmi_watchdog=1). > It is worth a try, however. With nmi_watchdog=1, I see the following output from "dmesg | grep -i nmi" with the 2.6.11.11 kernel I very recently installed: Bootdata ok (command line is root=/dev/sda3 vga=0x31a selinux=0 console=tty0 resume=/dev/sda2 desktop elevator=as nmi_watchdog=1 splash=verbose) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) Kernel command line: root=/dev/sda3 vga=0x31a selinux=0 console=tty0 resume=/dev/sda2 desktop elevator=as nmi_watchdog=1 splash=verbose activating NMI Watchdog ... done. testing NMI watchdog ... CPU#1: NMI appears to be stuck (0)! I'll try nmi_watchdog=2 shortly. Thanks, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCp3v8Hl0iXDssISsRAoDJAJ0TonNTSEAd+4W0GRPwTW/UoGFssACdEEq9 8IgEFJ7VAL/FVeyPXrEapHg= =jcjE -----END PGP SIGNATURE-----

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Michael Long

Andy Stewart wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >John Stoffel wrote: > > >>Andy, >> >>The best thing to do here is to run a serial console to another one of >>your systems and use THAT to try and catch the issue. You can then do >>a SysRq from the remote serial console and/or catch any error >>messages. >> >>Also, are there any messages in dmesg or /var/log/messages? How does >>the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see >>if the problem repeats. >> >>John >> >> > >HI John, > >I'm going to give the serial console a try - I may have a cable lying >around. > >I upgraded to the 2.6.11.11 kernel which did have several fixes for K8. > We'll see how that goes. > >Thanks for the ideas! > >Andy > >- -- >Andy Stewart, Founder >Worcester Linux Users' Group >Worcester, MA, USA >http://www.wlug.org > >-----BEGIN PGP SIGNATURE----- >Version: GnuPG v1.2.5 (GNU/Linux) >Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > >iD8DBQFCp2yhHl0iXDssISsRAqxcAJwLntOh9GpNmefW/+JdwbAUMRJKBACfbV1P >s2eqMUnY6rjZaIfI6MzZLqQ= >=aLfl >-----END PGP SIGNATURE----- >_______________________________________________ >Wlug mailing list >Wlug(a)mail.wlug.org >http://mail.wlug.org/mailman/listinfo/wlug > > Andy, If that doesn't help try using the system without the kvm and see if you get the same behavior. I have seen odd things in the past when using a faulty kvm switch. Mike

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 John Stoffel wrote: > Andy, > > The best thing to do here is to run a serial console to another one of > your systems and use THAT to try and catch the issue. You can then do > a SysRq from the remote serial console and/or catch any error > messages. > > Also, are there any messages in dmesg or /var/log/messages? How does > the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see > if the problem repeats. > > John HI John, I'm going to give the serial console a try - I may have a cable lying around. I upgraded to the 2.6.11.11 kernel which did have several fixes for K8. We'll see how that goes. Thanks for the ideas! Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA, USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCp2yhHl0iXDssISsRAqxcAJwLntOh9GpNmefW/+JdwbAUMRJKBACfbV1P s2eqMUnY6rjZaIfI6MzZLqQ= =aLfl -----END PGP SIGNATURE-----

June 8, 2005

Job opportunity in Natick for experienced Unix admin

by Simoncini, Matthew

List, I work for a large consulting company in Natick, MA. and we're looking for a couple of experienced (mid level) Unix administrators to join our team. Some of the skills that are required include: Unix experience (Sun Solaris 8 or higher) Veritas volume manager Veritas cluster services Storage management experience (preferably EMC Clarion or DMX) Backup management experience (preferably EMC EDM) Team player Desire to learn and grow If you or anyone you know is looking for a position as a Unix Administrator, please contact me off list at matthewsmbox-resumes(a)yahoo.com <mailto:matthewsmbox-resumes@yahoo.com> . Thanks. Matthew

June 8, 2005

MySQL curses management tool?

by John Stoffel

Hi all, A quick googling didn't help, so I'm turning to you all. I'm looking for a mysql management tool which is CLI/curses based and more powerful/easier to use when compared to the mysqlaccess and mysqladmin stuff. Basically, something to help me manage DBs, users and permissions over an SSH connection. I know I could use some of the X tools, but I'd prefer not too if possible. Thanks, John

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by John Stoffel

Andy, The best thing to do here is to run a serial console to another one of your systems and use THAT to try and catch the issue. You can then do a SysRq from the remote serial console and/or catch any error messages. Also, are there any messages in dmesg or /var/log/messages? How does the bootup look? Any warnings? Also, please try 2.6.12-rc6 and see if the problem repeats. John

June 8, 2005

Re: [Wlug] HOWTO debug hard lockups

by Jeff Moyer

==> Regarding Re: [Wlug] HOWTO debug hard lockups; Andy Stewart <andystewart(a)comcast.net> adds: andystewart> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 andystewart> Jeff Moyer wrote: >> ==> Regarding [Wlug] HOWTO debug hard lockups; Andy Stewart >> <andystewart(a)comcast.net> adds: >> andystewart> HI gang, >> andystewart> My dual Opteron machine is not happy. I cannot get more than andystewart> 7 straight days of uptime without getting a hard lock, andystewart> requiring a reboot. (My definition of hard lock is: machine andystewart> responds neither to keyboard input, mouse input, nor network andystewart> pings). >> andystewart> I can stimulate hard locks by running OpenOffice 1.1.3 (I had andystewart> 3 tonight, and 3-4 on a previous occasion while running andystewart> OpenOffice). It makes no sense to me that an application run andystewart> as a normal user could lockup a machine. >> andystewart> I've tried setting "nmi_watchdog=1" to see if I could get an andystewart> "oops" when it hard locks - no dice. Do you know any other andystewart> tricks I could try to see if it is the kernel which is locking andystewart> up? I'm running SuSE's version of 2.6.8. >> Did you verify that NMIs are being delivered? After boot, cat >> /proc/interrupts and make sure the NMI line is non-zero. Also note >> that, at least with upstream and Red Hat kernels, the nmi_watchdog >> defaults to 1 for Opterons (i.e. you shouldn't need to manually set it). andystewart> HI Jeff, andystewart> Well, this is weird, I'm seeing a ZERO count for NMIs, so that andystewart> makes me think they are NOT being delivered. How would I go andystewart> about solving *that* little problem? I did a "cat Well, you can try booting with nmi_watchdog=2. This will try to use the local APIC to deliver nmi's, but I haven't actually seen a dual processor system that required this (all of them I've seen work with nmi_watchdog=1). It is worth a try, however. andystewart> /proc/cmdline" to insure that I had "nmi_watchdog=1" and andystewart> indeed it is there. Perhaps this is the clue we've been andystewart> seeking. Well, it only tells you that the nmi_watchdog won't trigger. We still have no insight into what the problem might actually be. >> If the NMI watchdog works, it will print a message to the console. >> However, you will not see this if you are in X windows. Do you have a >> serial console hooked up, by any chance? I strongly suggest it if you >> have the means. andystewart> I think I have a cable to which I could connect the serial andystewart> port of the Opteron to the serial port of another Linux box andystewart> (and then use minicom or some such terminal program). >> Aside from this, if it is indeed a hard lockup, there is really nothing >> you can do (without purchasing other hardware to help debug the >> problem). andystewart> Yeah, I was afraid of that. >> Please give these suggestions a shot and let us know how it goes. andystewart> Shall do - thanks, everybody! andystewart> Oh, be advised that when you smash your fist on the keyboard andystewart> after your system locks up for the umpteenth time, that a lot andystewart> of dead skin cells will come flying upward out of the bowels andystewart> of the keyboard. I would recommend safety glasses. Noted. =) -Jeff

June 8, 2005